Loading...
Loading...
The PowerProtect Data Manager is used to protect the vSphere environment with the Transparent Snapshot Data Mover (TSDM) solution. The virtual machine (VM) backups fail with exit status ABV0016 during the "Creating LWD snapshot" step. The PowerProtect Data Manager Interface job details show: ABV0016: VM Direct engine 'localhost' is unable to back up the virtual machine '<my-vm>' on vCenter 'my-vcenter.domain.local' because of a vCenter API issue.The VM Direct engine cannot complete the backup of this virtual machine because a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read LesstaskInfo.Error.Fault = '*types.InvalidState', &{VimFault:{MethodFault:{FaultCause:<nil> FaultMessage:[]}}} The vSphere Web Client shows tasks for the affected VM shows continuous error tasks when taking the LWD-based snapshot. The backup export log shows: YYYY-MM-DD HH:MM:SS DEBUG: [a2aaedf09233d1a7;b0db985f58505615] SDM Data Mover: Create LWD Snapshot: vCenter SOAP call 'DpSnapshot' was accepted. YYYY-MM-DD HH:MM:SS DEBUG: [a2aaedf09233d1a7;b0db985f58505615] SDM Data Mover: Create LWD Snapshot: taskInfo.Error.Fault = '*types.InvalidState', &{VimFault:{MethodFault:{FaultCause:<nil> FaultMessage:[]}}} YYYY-MM-DD HH:MM:SS ERROR: [a2aaedf09233d1a7;b0db985f58505615] SDM Data Mover: Create LWD Snapshot: vCenter task 'task-1960194' failed. YYYY-MM-DD HH:MM:SS ERROR: [a2aaedf09233d1a7;b0db985f58505615] SDM Data Mover: Create LWD Snapshot: Unable to create LWD snapshot. YYYY-MM-DD HH:MM:SS DEBUG: [a2aaedf09233d1a7;b0db985f58505615] SDM Data Mover: Create LWD Snapshot: Retrying in 15s... The ESXi host tsdm.log shows a continuous failure when trying to communicate with the Data Protection Daemon (DPD). YYYY-MM-DD HH:MM:SS tsdm[2099213]: IPCMGR: Unable to connect to UDS at /var/run/vmwarelwd/daemon: No such file or directory YYYY-MM-DD HH:MM:SS tsdm[2099213]: IPCMGR: Error creating a connection to add to pool 'dpd': No such file or directory
The DPD is an invalid state on the ESXi server hosting the affected VMs. The vSphere Web Client shows the DPD daemon in a Stopped status. The cause of the DPD being invalid and showing a Stopped state is unknown. The ESXi host dpd.log shows an exception a few days ago and then stopped logging. YYYY-MM-DD HH:MM:SS error dpd[2147513] [Originator@6876 sub=vmomi.soapStub[0]] Resetting stub adapter: service state request failed; /sdk/vimService, <last binding: <<TCP '127.0.0.1 : 31138'>, <TCP '127.0.0.1 : 80'>>>, N7Vmacore15SystemExceptionE(Host not found (authoritative)) --> [context]zKq7AVICAgAAABAEVQEOZHBkAAAtoUJsaWJ2bWFjb3JlLnNvAADiZx0ALdEbANJCFwAcoC0AwG8zAD9zMwDhTTQAy1wuAMzHLQBUAy4AAhE/ATt9AGxpYnB0aHJlYWQuc28uMAACbdEObGliYy5zby42AA==[/context]
The DPD must be restarted on the affected ESXi host. The DPD may be started using an ESXi ssh session: [root@my-esx:~] /etc/init.d/dpd status dpd is stopped [root@my-esx:~] /etc/init.d/dpd start Starting dpd:Setting the memory limit for dpd resource pool on this host to 320 MB Setting max number of file descriptors to 800 Setting MALLOC_ARENA_MAX to 20 success[root@my-esx:~] /etc/init.d/dpd status dpd is running The DPD may also be started using the vSphere Web Client for the affected ESXi host under " Configure > Services ": If required, contact VMware Support to further investigate the DPD being invalid and showing a Stopped state.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.