Symptoms
To troubleshoot this issue, assuming no backup is holding the files, use VMware's KB here to check if any ESXi server is locking a file:https://kb.vmware.com/s/article/2107795 Note: This VMware KB article is not targeted for vSAN environments. While lock file manipulation and behaviors are universal, this KB contains verbiage and methodologies pertaining to nonvSAN storage such as VMFS and NFS. Practice extreme caution and ensure that valid backups exist when manipulating files manually in a VxRail environment.Try also:
Restart the management agents on the nodes that owned this VM: "services.sh restart"vMotion the VM to another ESXi host.
VxRail: Unable to consolidate snapshots. Unable to see lock on one of the VM files.Consolidate virtual machine disk files: Unable to access file since it is locked.See the error stack for details on the cause of this problem:Time: 6/6/2018 11:12:51 PMTarget: SERVERNamevCenter Server: vCenter.Name.localError StackAn error occurred while consolidating disks: msg.fileio.lock.Consolidation failed for disk node 'scsi1:0': msg.fileio.lock.
Log in as root to the ESXi host where the affected virtual machine is running through SSH.Go to /vmfs/volumes/vm-datastore/vm-dir/. Open the /var/log/vmware.log file and look at the last consolidate/power-on process.
vmware.log
2018-06-07T05:15:33.039Z| vcpu-0| I125: DISK: OPEN scsi1:0 '/vmfs/volumes/vsan:52279f9f4c6ba5cd-d5fbf7f705ba65f4/382ad059-6ebb-eabe-80f2-246e9668e1b8/SERVERNAME-000003.vmdk' persistent R[]
2018-06-07T05:15:33.044Z| vcpu-0| I125: DISKLIB-VMFS : "vsan://9c3c115b-fd7e-6737-c06b-246e9668e290" : open successful (10) size = 1099511627776, hd = 3154212736. Type 19
......................
......................
2018-06-07T05:15:33.077Z| vcpu-0| I125: VigorTransport_ServerSendResponse opID=4DDDC7A4-000006F8-df-a6-3861 seq=3793: Completed Snapshot request.
2018-06-07T05:15:33.077Z| vcpu-0| I125: Turning off snapshot info cache.
2018-06-07T05:15:33.079Z| vcpu-0| I125: ConsolidateEnd: Snapshot consolidate complete: Failed to lock the file (5).
Cause
User was running a Veeam backup on these VMs. Even though it was not running a backup then, this was still holding some lock on the VM files.
Resolution
Veeam backup was holding the vmdk.After removing the VMDK from Veeam, the disk consolidation was completed.
