OPERATIONAL DEFECT DATABASE
...
...
Snapshots cannot be committed.Third-party backup application (Veeam, Avamar, Unitrends,...etc) has locked snapshot files, and failed to remove the snapshot after completing backups or failed to initiate backups. Snapshot commits without errors and Snapshot Manager is no longer populated. However, the snapshot disk is present in the virtual machine directory. (the VM is still running on Snapshot)The virtual machine summary tab display messages similar to:Snapshot consolidation required.Virtual machine disks consolidation is needed.Virtual machine Consolidation Needed status. Consolidation failed with the errors similar to:Failed to lock the file or One or more disks are busy.Unable to consolidate virtual machine snapshots due to file lock.Unable to access filename since it is locked.Unable to access file since it is locked In hostd.log on the ESXi Host where the virtual machine is running, you see entries similar to the below:Note: hostd.log can be located in /var/run/log/hostd.log2021-07-08T17:27:12.819Z info hostd[1050487] [Originator@6876 sub=Vimsvc.ha-eventmgr] Event 361 : Lock(s) held on file Win-test-flat.vmdk by other host(s).2021-07-08T17:27:12.822Z info hostd[1051013] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-662492021-07-08T17:27:12.823Z info hostd[1050756] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb069 user=vpxuser] Task Completed : haTask--vim.event.EventHistoryCollector.readNext-66249 Status success2021-07-08T17:27:12.826Z info hostd[1050487] [Originator@6876 sub=Vimsvc.TaskManager opID=d88eb06c user=vpxuser] Task Created : haTask--vim.event.EventHistoryCollector.readNext-66250 Can’t power on virtual machine due to file lock:Unable to open Swap File.Unable to access a file since it is locked.Unable to access a file filename since it is locked.Unable to access Virtual machine configuration.An unexpected error was received from the ESX host while powering on VM <VM_name>. Reason: Cannot open the disk disk_name or one of the snapshot disks it depends on. In vmkernel.log , on the ESXi Host where the virtual machine is running, you see entries similar to the below: Note: vmkernel.log can be located in /var/run/log/vmkernel.log WARNING: World: VM xxxx: xxx: Failed to open swap file path: Lock was not freeWARNING: World: VM xxxx: xxx: Failed to initialize swap file path
When a virtual machine snapshot fails to consolidate or a virtual machine with snapshots fails to power on, it is usually because one or more of the virtual machine files are abnormally locked by the same ESXi Host the virtual machine is running or any other ESXi Host.The ESXi Host (either same Host where the VM is running or any other Host) is abnormally locking the virtual machine files due to many reasons and not limited to the below: In-completed or interrupted backup taskPrevious storage failure
Caution: Make sure there is no backup or snapshot task running on the virtual machine before applying the resolution.
Virtual Machine File Lock most common scenarios:1-Proxy Back Up Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files.2-An ESXi Host is locking the Virtual Machine files abnormally.First Scenario The most common case of virtual machine file lock is when the Backup Proxy Virtual Machine (Veeam, Avamar, Unitrends,...etc) is locking one or more of the virtual machine .vmdk files. This issue can be resolved using the below steps: Right-click the backup proxy virtual machine.Click Edit Settings.Expand all the Hard Disk(s).Select the Hard Disk(s) which belong to the virtual machine that has the problem.Click on the X beside the Hard Disk to unmount the Hard Disk from the VM. Caution: Do NOT select Delete files from the datastore. Click OK.Consolidate/Delete the snapshot on the VM. " Power on the VM if the issue was unable to power on" Second Scenario If the first section is not the case, then we need to identify which process is locking the virtual machine files to release. Connect to the ESXi host the virtual machine is on with an SSH sessionNavigate to the VM directory and take a note of where the VM is currently writing. To know the VM directory, follow the below steps:1- Run the below command# vim-cmd vmsvc/getallvms | grep -i "VM_Name"2- Run the below command:# cd /vmfs/volumes/datastore_name/virtual_machine_name/ --->From step 1 3-Run the below command:# grep -i vmdk *vmx*Now the VM is writing on Test_File_Lock-000002.vmdk in the example Check which ESXi Host is locking the VM file(s) and clear the lock by running the below script in the VM directory: # ls | while read x; do vmfsfilelockinfo -p $x| grep -i "is locked"; doneNote: The script will not work if the virtual machine files has spaces. In this case, you will need to run the command vmfsfilelockinfo -p on each and every file listed under the virtual machine directory.Example: vmfsfilelockinfo -p Test_File_Lock-flat.vmdk There will be two probabilities: (after running the script) The VM files are locked by same ESXi Host where the VM is running (owning Host). Notes:1-If the VM is powered on so it should be normally locked by the ESXi Host it is running on (referred by the MAC address in the script output) 2-If the VM is powered off then there shouldn't be an output from the script as VM files shouldn't be normally locked by any process (from any ESXi Host)To know what is locking the VM files and release the lock , follow the below : # lsof | grep -i "VM_Name"a- It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:# /etc/init.d/vpxa stop# /etc/init.d/vpxa startb- It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:# /etc/init.d/hostd stop# /etc/init.d/hostd startc- The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files#cd /dev/deltadisks#rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss.Notes: If the VM is powered off then, this script should show no output as no process should be locking the VM files. So if the script shows and output and the VM is powered off, then there is a stale process that will only be released by rebooting the ESXi Host. If you got the error "device or resource busy" while trying to run the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/detadisks and perform the remove again. The VM files are locked by another ESXi Host All the files are locked by Host that has the same MAC address except one file "Test_File_Lock-flat.vmdk" that is locked by two ESXi Hosts (two MAC addresses)As the VM is powered on so there should be normal lock from the owing ESXi Host but there is another abnormal lock coming from another Host that has the MAC address "00:50:56:01:81:e8" in the example To release the lock from the other ESXi: Identify which ESXi Host locking the the file from the vCenter GUI Expand the Cluster and select any ESXi HostClick on ConfigureClick on Physical adapters and look for the MAC address "Output from the previous step" Note: You can identify which ESXi Host has a NIC with the MAC address resulted from the script using the below command: (only if vSphere High Availability is enabled)# esxcli network ip neighbor listOutput should be like the below:Neighbor Mac Address Vmknic Expiry State Type-------------- ----------------- ------ -------- ----- -------192.168.0.10 00:50:56:01:30:09 vmk0 1190 sec Unknown192.168.0.1 00:50:56:01:b0:01 vmk0 989 sec Unknown192.168.0.82 00:50:56:01:30:16 vmk0 572 sec Unknown ----> 192.168.0.82 is the IP address of the locking ESXi Host192.168.0.51 00:50:56:01:30:0a vmk0 662 sec Unknown192.168.0.83 00:50:56:01:30:1c vmk0 733 sec Unknown192.168.10.120 00:50:56:01:01:09 vmk1 122 sec Unknown192.168.20.82 00:50:56:6f:2c:c6 vmk2 833 sec Unknown Connect to the ESXi Host where the virtual machine is on with an SSH sessionTo know what is locking the VM files and release the lock , follow the below : # lsof | grep -i "VM_Name" a- It could be vpxa-worker ----> this can be fixed by restart vpxa agent using the below commands:# /etc/init.d/vpxa stop# /etc/init.d/vpxa startb- It could be hostd or hostd-profiler ---> this can be fixed by restarting hostd agent using the below commands:# /etc/init.d/hostd stop# /etc/init.d/hostd startc- The VM files might be stored in a temp directory /dev/deltadisks----> this can be fixed by going to the temp directory and remove the files#cd /dev/deltadisks#rm -rf "VM_Name_00000x-delta.vmdk" ---->file name might be slightly different like "8857142-Test_File_Lock-000002-delta.vmdk"Warning: Make sure that the directory is /dev/deltadisks . This is the only directory where it is safe to remove the VM files without data loss. Notes: If the VM has no clear process locking the files (no output from the command lsof | grep -i "VM_Name") then this is a stale process and the locking ESXi Host has to be rebooted to release the lock.If you got the error "device or resource busy" while trying to run the rm command , you will need to power off the VM, make sure that there is still a VM file in the /dev/detadisks and perform the remove again.
For more information , see Troubleshooting issues resulting from locked virtual disksFor more information about Virtual Machine File Lock, see Investigating virtual machine file locks on ESXi hosts
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
