Loading...
Loading...
The following conditions may trigger a timeout operation within vCenter and require adjusting some vCenter Server advanced settings in order to workaround the problem. Some criteria that can trigger this behavior is one or more of the following: · Relocating a virtual machine from one ESXi host to another (Powered-Off) · In addition to changing ESXi hosts, changing datastore types (in some cases across array types) · The virtual machine disk size is larger than 1TB · The disk type will change during the migration (ie. Changing from thin provisioned to Eager Zeroed Thick) The error observed in vCenter Tasks will be “Operation timed out” Within the vCenter Server (vpxd.log) logs, you may see: 2020-04-01T07:42:23.996Z info vpxd[01615] [Originator@6876 sub=vpxTaskInfo] Timed out waiting for task vim.Task:task-57792020-04-01T07:42:23.996Z warning vpxd[01615] [Originator@6876 sub=vpxLro opID=sroger5str-279239-auto-5zgo-h5:70021205-32-02] [VpxLRO] Timeout waiting on updates for task-57792020-04-01T07:42:23.997Z warning vpxd[01615] [Originator@6876 sub=vpxLro opID=sroger5str-279239-auto-5zgo-h5:70021205-32-02] [VpxLRO] Timeout waiting on updates for task-57792020-04-01T07:42:23.997Z error vpxd[01660] [Originator@6876 sub=VmProv opID=sroger5str-279239-auto-5zgo-h5:70021205-32-02] Get exception while executing action vpx.vmprov.CopyVmFiles: N3Vim5Fault8Timedout9ExceptionE(Fault cause: vim.fault.Timedout--> ) On the source ESXi host, vpxa.log would note the disk copy starting successfully: 2020-04-01T03:44:09.361Z verbose vpxa[2099607] [Originator@6876 sub=NfcManager opID=sroger5str-210840-auto-4iop-h5:70017269-5-02-5c] Transfer file /vmfs/volumes/5505ede1-3c37e4a8-c061-0025b5020430/monster_VM/monster_VM_1.vmdk -> /vmfs/volumes/5e83a485-8b447362-3181-0025b5a201f8/monster_VM/monster_VM_1.vmdk (parent: , size: 1073741824000, flags: 4259912, grainSize: 0 policy: '(null)') Approximately 30 minutes later, the destination ESXi host logs (hostd.log) would mark the operation failed and vCenter would log the timeout: hostd.0:2020-04-01T04:23:59.783Z warning hostd[2099211] [Originator@6876 sub=Libs opID=sroger5str-210840-auto-4iop-h5:70017269-5-02-f6-7c8e user=vpxuser:VMWARE\jsmith] Hostlog_ExecuteCleanup: Failed to stat disk /vmfs/volumes/5e83a485-8b447362-3181-0025b5a201f8/monster_VM/monster_VM_1.vmdk - not cleaning up. Hlog: /vmfs/volumes/5e83a485-8b447362-3181-0025b5a201f8/monster_VM/monster_VM-1e67cef9.hlog
In some cases, based on the scenario at hand, the NFC copy operation may take a long time and could hit the upper threshold of some of vCenter Server's timeout settings. This can contribute to relocate operations timing out for large virtual machines.This article is intended to inform of what settings to change to ensure the relocate operation can complete successfully.
This is caused due to the default timeout for vCenter Server for the following advanced settings: config.task.timeout config.vmomi.soapStubAdapter.blockingTimeoutSeconds The timeout is hit in specific cases where: · A large virtual machine disk size is being copied – upwards of 1TB or larger · The relocate operation leverages NFC as the virtual machine is powered off and copying across two different VMFS datastores (in some cases different arrays) · The source and destination ESXi host management networks are in different broadcast domains · The disk type for the virtual machine is changed during the copy (ie. Converting from thin to thick provisioned VMDK’s)
A restart of the vCenter Server process (vpxd) will be required in order for the resolution instructions to take effect.
If relocating the virtual machine (leveraging NFC) is the only method available, the following advanced settings would need to be added in vCenter: Name: config.task.timeout Value: 7200 Name: config.vmomi.soapStubAdapter.blockingTimeoutSeconds Value: 18000 Note: The advanced settings do not exist in vCenter and must be added. The default value is 1800 seconds (30min) for both. VMware recommends that you change these settings only when instructed to do so by VMware technical support or when you are following specific instructions in VMware documentation. A restart of vCenter Server is required for the settings to take effect.
For VMware documentation on vCenter Server advanced settings, and how to insert custom settings, please refer to the documentation below: https://docs.vmware.com/en/VMware-vSphere/6.5/com.vmware.vsphere.vcenterhost.doc/GUID-62184858-32E6-4DAC-A700-2C80667C3558.html https://docs.vmware.com/en/VMware-vSphere/6.7/com.vmware.vsphere.vcenterhost.doc/GUID-62184858-32E6-4DAC-A700-2C80667C3558.html
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.