OPERATIONAL DEFECT DATABASE
...
...
After performing a Storage vMotion of a Windows 2008 or 2008 R2 Virtual machine on an ESX/ESXi 4.1u1 and 5.0 host you experience the following: Attempting to backup or snapshot the Virtual machine fails with the error:Cannot create a quiesced snapshot because the create snapshot operation exceeded the time limit for holding off I/O in the frozen virtual machine Taking a manual snapshot is successful when including both snapshot the Virtual machine's memory and quiesce guest file system, but fails when only quiesce guest file system is selected. The hostd.log file may show errors similar to:F5909B90 error 'vm:/vmfs/volumes/datatore/virtual_machine/virtual_machine.vmx' opID=2c7d0baf-f9] Quiesced snapshot backup agent error: Asynchronous operation failed: VssSyncStart (5)message = "Asynchronous operation failed: VssSyncStart",F5548B90 error 'vm:/vmfs/volumes/datatore/virtual_machine/virtual_machine.vmx' opID=83a9f34b-8a] Quiesced snapshot backup agent error: Asynchronous operation failed: VssSyncStart (5)message = "Asynchronous operation failed: VssSyncStart", The vmware.log file may show errors similar to:DISKLIB-VMFS : "/vmfs/volumes/datatore/virtual_machine/virtual_machine-flat.vmdk" : open successful (21) size = 53687091200, hd = 0. Type 3DISKLIB-VMFS : "/vmfs/volumes/datatore/virtual_machine/virtual_machine-flat.vmdk" : closed.DISKLIB-VMFS : "/vmfs/volumes/datatore/virtual_machine/virtual_machine-flat.vmdk" : open successful (21) size = 64424509440, hd = 0. Type 3DISKLIB-VMFS : "/vmfs/volumes/datatore/virtual_machine/virtual_machine-flat.vmdk" : closed.SOCKET 1 (99) recv detected client closed connectionVix: [202112 mainDispatch.c:2472]: VMAutomation: Connection Error (4) on connection 0.VmdbPipeStreamsOvlError Couldn't read: OVL_STATUS_EOF If you are enable VMware Tools in debug mode, the vmware.log file of the virtual machine reports errors similar to:[10572] [debug] [vmbackup] *** CVmVssRequestor::QueryStatus[8092] SnapRequestor: CVmSnapshotRequestor::QueryStatus[8092] VssProvider_IsDiskSupported: vendor id is null.[8092] VDSHelper::ForEachVolume: return (0)[8092] CVmSnapshotRequestor::AddSupportedVolumes: return (8004230c) <---- search for this number[8092] CVmSnapshotRequestor::PrepareBackup():1080: failed call: ret = AddSupportedVolumes(), result = 8004230c[8092] SnapRequestor: CVmSnapshotRequestor::Finalize[8092] SnapRequestor: CVmSnapshotRequestor::UnregisterProvider[8092] SnapRequestor: Provider unregistered.[10572] [ debug] [vmbackup] *** VmBackupAsyncCallback[10572] [ debug] [vmbackup] VmBackupAsyncCallback: checking VssSyncStart[10572] [ debug] [vmbackup] *** VmVssBackupOpQuery[10572] [ debug] [vmbackup] *** CVmVssRequestor::QueryStatus[8092] SnapRequestor: CVmSnapshotRequestor::QueryStatus[10572] [ debug] [vmbackup] Async operation error: 0x8004230c[10572] [ debug] [vmbackup] *** VmBackup_SendEvent[10572] [ debug] [vmbackup] *** VmVssBackupOpReleaseFor more information on enabling VMware Tools in debug mode, see Enabling debug logging for VMware Tools within a guest operating system (1007873). When checking the virtual machine disk properties using PowerShell you see that the Serial Number for the disk is blank:C:\> Get-WmiObject -ComputerName <VM_hostname> Win32_DiskDrive -credential <domain>\<user> | Format-List -property *The serial number value is missing from the output:VM with disk.enableUUID = "TRUE"SCSITargetId : 0SerialNumber : <-------- The disk UUID/Serial number is blankSignature : 357258723 In the Application Event logs of the Event Viewer within the Windows guest operating system, you see the error:"Volume Shadow Copy Service error: Unexpected error DeviceIoControl" Operation:Exposing Recovered VolumesLocating shadow-copy LUNsPostSnapshot EventExecuting Asynchronous Operation Context:Device: \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}Examining Detected Volume: Existing - \\?\fdc#generic_floppy_drive#6&2bc13940&0&0#{53f5630d-b6bf-11d0-94f2-00a0c91efb8b}Execution Context: ProviderProvider Name: VMware Snapshot ProviderProvider Version: 1.0.0Provider ID: {564d7761-7265-2056-5353-2050726f7669}Current State: DoSnapshotSet The Tasks & Events tab of the vSphere Client reports the error:'VssSyncStart' operation failed: IDispatch error #8460 (0x8004230c)
This issue is caused when the Microsoft VSS service is attempting to do an application quiesced snapshot but is unable to read the disk UUID or Serial Number value for one or more virtual machine disks. For more information on application quiesced snapshots, see Enabling Windows 2008 application-consistent quiescing on ESX/ESXi 4.1 (1028881). With ESX/ESXi 4.1 Update 1 and later, support for application quiesced snapshots for Windows 2008 and 2008 R2 Virtual machines was introduced. Application quiesced snapshot support required the new disk.enableUUID parameter to be added to the Virtual machine's .vmx configuration file to enable or disable the feature. When the parameter has been enabled, you need to reboot the virtual machine to ensure that the disk UUID/Serial Number is also populated as it is required for application snapshots. The disk.enableUUID parameter did not exist on earlier versions of ESX/ESXi, therefore virtual machines created using those earlier versions will not have that parameter included in their .vmx configuration file. If the ESX/ESXi hosts are upgraded from an earlier version to ESX/ESXi 4.1, support for application quiesced snapshots will be available. When a Virtual machine is migrated to a new datastore using Storage vMotion, the disk.enableUUID parameter is added to the Virtual machine's .vmx configuration file with a value of true if the parameter is not already present in the file. Before the migration, the effective value of disk.enableUUID is false (as the feature is not being used), thus the migration is changing the default value from false to true. When a backup or snapshot attempts to quiesce the Virtual machine, it will see that application quiescing has been enabled (disk.enableUUID=true), but it will be unable to read the disk UUID/Serial Number as the virtual machine has not been rebooted since that parameter was added to the .vmx configuration file.
This issue is resolved in ESX/ESXi 4.1 Update 3, ESXi 5.0 Update 2 and ESXi 5.1 Update 1. For further details see:ESXi 4.1 U3 Release Notes - Resolved IssuesESXi 5.0 U2 Release Notes - Resolved IssuesESXi 5.1 U1 Release Notes - Resolved IssuesTo work around this issue in earlier builds of ESX/ESXi, use one of these options: Allow VSS application quiescing to work by generating Disk UUID/Serial number. This can be achieved by leaving the disk.enableUUID value to true and power cycle of the virtual machine. Disable VSS application quiescing feature by setting one of these options.Note: This is a non-disruptive process and does not require Virtual machine downtime. Disable application quiescing by disabling the option from VMware Tools. In the Windows guest, create the file %SystemDrive%\ProgramData\VMware\VMware Tools\tools.conf. Add these entries to the tools.conf file:[vmbackup]vss.disableAppQuiescing = true Restart the VMware Tools Service. Change the value of the disk.enableUUID parameter from true to false by modifying the virtual machine's configuration file (vmname.vmx) and migrate the virtual machine to another host in the environment to reload the virtual machine configuration into the host memory.Note: You may use PowerCLI scripts to modify the value on multiple virtual machines simultaneously. VMware Support does not provide PowerCLI scripts under standard support agreement and, therefore, you may have to refer the PowerCLI user guide for creating your own script or contact VMware SDK support.
Enabling debug logging for VMware Tools within a guest operating systemEnabling and disabling Windows 2008 and later application-consistent quiescing on ESXi/ESX ESX/ESXi 4.1 または 5.0 ホスト上で Storage vMotion が失敗した後に仮想マシンのスナップショットを取得する在 ESX/ESXi 4.1 或 5.0 主机上执行 Storage vMotion 后创建虚拟机快照失败
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
