OPERATIONAL DEFECT DATABASE
...
...
Powering on the virtual machine results in the power on task remaining at 95% indefinitely.Adding an existing virtual machine disk (VMDK) to a powered on virtual machine fails with the error: Failed to add disk scsi0:1. Failed to power on scsi0:1Cannot power on the virtual machine after deploying it from a template.Powering on a virtual machine fails with an error:Unable to open Swap FileUnable to access a file since it is lockedUnable to access a file filename since it is lockedUnable to access Virtual machine configuration In the /var/log/vmkernel log file, you see entries similar to: WARNING: World: VM xxxx: xxx: Failed to open swap file path: Lock was not freeWARNING: World: VM xxxx: xxx: Failed to initialize swap file path When opening a console to the virtual machine, you may receive the error: Error connecting to path/virtual machine/vmx because the VMX is not started The virtual machine reports conflicting power states between vCenter Server and the ESXi host UI.Attempting to view or open the .vmx file using a text editor (ex: cat or vi), reports an error similar to: cat: can't open '[name of vm].vmx': Invalid argument
The Purpose of File Locking To prevent concurrent changes to critical virtual machine files and file systems, ESXi hosts establish locks on these files. In certain circumstances, these locks may not be released when the virtual machine is powered off. As such, the files cannot then be accessed by other ESXi hosts while locked and the virtual machine fails to power on.Virtual machine files locked during runtime include: VMNAME.vswpDISKNAME-flat.vmdkDISKNAME-ITERATION-delta.vmdkVMNAME.vmxVMNAME.vmxfvmware.log Initial Quick Test Set DRS in manual mode and attempt to power on the virtual machine on all hosts in the cluster. If the virtual machine powers on, that host help the lock. Identifying the Locked File Power on the virtual machine, this process should fail and display an error message. Make note of the this error.Connect to the ESXi host the virtual machine is on with an SSH sessionFind the IP address of the host holding the lock by running vmfsfilelockinfo on the VMDK flat, delta, or sesparse file for VMFS, or the .UUID.lck file for vSAN. vmfsfilelockinfo takes these parameters: File to be testedUsername and password for accessing VMware vCenter Server (when tracing MAC address to ESX host.) For example:vmfsfilelockinfo -p /vmfs/volumes/iscsi-lefthand-2/VM1/VM1_1-000001-delta.vmdk -v 192.168.1.10 -u administrator@vsphere.localvmfsflelockinfo Version 1.0Looking for lock owners on "VM1_1-000001-delta.vmdk""VM1_1-000001-delta.vmdk" is locked in Exclusive mode by host having mac address ['xx:xx:xx:xx:xx:xx']Trying to make use of Fault Domain Manager----------------------------------------------------------------------Found 0 ESX hosts using Fault Domain Manager.----------------------------------------------------------------------Could not get information from Fault domain managerConnecting to 192.168.1.10 with user administrator@vsphere.localPassword: xXxXxXxXxXx----------------------------------------------------------------------Found 3 ESX hosts from Virtual Center Server.----------------------------------------------------------------------Searching on Host 192.168.1.178Searching on Host 192.168.1.179Searching on Host 192.168.1.180MAC Address : xx:xx:xx:xx:xx:xxHost owning the lock on the vmdk is 192.168.1.180, lockMode : ExclusiveTotal time taken : 0.27 seconds.Note: During the life-cycle of a powered on virtual machine, several of its files transitions between various legitimate lock states. The lock state mode indicates the type of lock that is on the file. The list of lock modes is: mode 0 = no lockmode 1 = is an exclusive lock (vmx file of a powered on virtual machine, the currently used disk (flat or delta), *vswp, and so on.)mode 2 = is a read-only lock (For example on the ..-flat.vmdk of a running virtual machine with snapshots)mode 3 = is a multi-writer lock (For example used for MSCS clusters disks or FT VMs) To find the name of the process holding the lock, run lsof 'locked_file_name' on the host holding the lock and filter the output for the file name in question: For example: lsof | egrep 'Cartel|VM1_1-000001-delta.vmdk'Cartel | World name | Type | fd | Description36202 vmx FILE 80 /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/VM1/VM1_1-000001-delta.vmdkThis shows that the file is locked by a virtual machine having Cartel ID 36202. Now display the list of active Cartel IDs by with this command: esxcli vm process listThis displays information for active virtual machines grouped by virtual machine name and having a format similar to:Alternate_VM27World ID: 36205Process ID: 0VMX Cartel ID: 36202UUID: 56 4d bd a1 1d 10 98 0f-c1 41 85 ea a9 dc 9f bfDisplay Name: Alternate_VM27Config File: /vmfs/volumes/556ce175-7f7bed3f-eb72-000c2998c47d/Alternate_VM27/Alternate_VM27.vmx………The virtual machine entry having VMX Cartel ID: 36202 shows the display name of the virtual machine holding the lock on file VM1_1-000001-delta.vmdk, which in this example, is Alternate_VM27. If no processes are shown, the following script can search for VMs with the locked vmdk mounted. for i in $(vim-cmd vmsvc/getallvms | grep -v Vmid | awk -F "/" '{print $2}' | awk '{print $1}'); do echo $i && find ./ -iname $i | xargs grep vmdk | grep -Ei VMDKS_TO_LOOK_FOR ; done Removing the Lock Power off the virtual machine holding the lockReboot the ESXi host that holds the lock Removing the .lck file (NFS only) Power down the virtual machine and remove lck-#### files in the virtual machines directory. Caution: Only remove the lck files of a powered off virtual machineNote: VMFS volumes do not have lck files. The locking mechanism for VMFS volumes is handled within VMFS metadata on the volume. Verify the integrity of the virtual machine configuration file (.vmx)For more information on checking the integrity of the virtual machine configuration file, see Verifying ESX/ESXi virtual machine file integrityFor related information, see "Cannot open the disk" errors powering on a VM ."An error occurred while consolidating disks:" deleting one or more snapshots or consolidate disksPowering off an unresponsive virtual machine on an ESXi host Opening a Support RequestIf the problem persists after completing the steps in this article: Collecting diagnostic information for VMware products .Investigating virtual machine file locks on ESXi
VMware Skyline Health Diagnostics for vSphere - FAQTroubleshooting virtual machine power on task hangsFailed to power on virtual machine
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
