Symptoms

The PowerProtect Data Manager is used to protect the vSphere environment with the Transparent Snapshot Data Mover (TSDM) solution. A few VM Asset backups started to fail, when they were completing successfully in the past. The failure details indicate the vCenter task fails with a TSDM error indicating "DPD Sync in progress." The ABV0016 and ABV0006 details show: ABV0016: VM Direct engine 'localhost' is unable to back up the virtual machine 'my-vm-name' on vCenter 'my-vcernter-name' because of a vCenter API issue. The VM Direct engine cannot complete the backup of this virtual machine because of a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read Less Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'my-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress. ABV0006: VM Direct engine 'localhost' is unable to back up the virtual machine 'my-vm-name' on vCenter 'my-vcernter-name' because of a vCenter API issue. The VM Direct engine cannot complete the backup of this virtual machine because of a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read Less vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress The Backup session may show the following status outputs: ... YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] }, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Status": { YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "State": "Failure", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Description": "Backup is unsuccessful: Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'vm-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Progress": 0, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Error": { YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Text": "Unable to cleanup from earlier backup attempt. Unable to cleanup session 'Backup:6755cb99-3de5-4731-8b6f-6c64bb809f0b' from virtual machine 'vm-vm-name' (MORef = 'vm-241'). Unable to cancel TSDM / LWD snapshot. The vCenter task for 'Cancel SDM Snapshot' completed with state 'error': DPD Sync in progress", YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Code": 0, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Classifications": [], YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Retryable": false, YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] "Remedy": null YYYY-MM-DD HH:MM:SS INFO: [acae20b6ae1ca9c8;aac63ea8d0fb1d9e] }, ... ... YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] }, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Status": { YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "State": "Failure", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Description": "Backup is unsuccessful: vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress, TSDM snapshot UUID = '52031978-18dd-444a-37fd-9c070de84cc5'", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Progress": 0, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Error": { YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Text": "vCenter task 'task-943698' failed: VimFault: 'dp.vpx.fault.SnapshotInProgress.summary': DPD Snapshot in progress, TSDM snapshot UUID = '52031978-18dd-444a-37fd-9c070de84cc5'", YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Code": 0, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Classifications": [], YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Retryable": false, YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] "Remedy": null YYYY-MM-DD HH:MM:SS INFO: [a3f45e322add25ec;bb6e18b9d1e8178c] }, ...

Cause

The affected VM has an orphaned vSphere Snapshot Data Mover (SDM) task that the TSDM backup workflow cannot cancel due to inconsistencies in the dpd daemon. The reason for the orphaned task or dpd inconsistency remains unidentified.

Resolution

The dpd daemon should be restarted on the ESXi host, and the SDM feature should be disabled and enabled on the affected VM. Ensure that there are no virtual machine backups before restarting the dpd daemon: Open an ssh session to the ESXi host.Restart the dpd daemon: /etc/init.d/dpd restart [root@my-server ~]# ssh root@my-esx The authenticity of host 'my-esx (192.168.9.111)' can't be established. ECDSA key fingerprint is SHA256:LHQVSVzDSw7ezlDVG59nyGD+C/Ugcx/jaSLMbzsY18U. This key is not known by any other names Are you sure you want to continue connecting (yes/no/[fingerprint])? yes Warning: Permanently added 'my-esx' (ECDSA) to the list of known hosts. (root@my-esx) Password: The time and date of this login have been sent to the system logs. WARNING: All commands run on the ESXi shell are logged and may be included in support bundles. Do not provide passwords directly on the command line. Most tools can prompt for secrets or accept them from standard input. VMware offers powerful and supported automation tools. Please see https://developer.vmware.com for details. The ESXi Shell can be disabled by an administrative user. See the vSphere Security documentation for more information. [root@my-esx:~] /etc/init.d/dpd restart Restarting dpd: dpd stopped. Setting the memory limit for dpd resource pool on this host to 320 MB Setting max number of file descriptors to 32768 Setting MALLOC_ARENA_MAX to 20 success [root@my-esx:~] Here are some of the available methods to disable and enabled the SDM on the affected VM. Method 1 Change the affected VM asset protection mechanism. Open a browser session to PowerProtect Data Manager and authenticate with an admin account.Browse to Infrastructure > Assets > Virtual Machines.Select the affected virtual machine.Select the dropdown More Actions > Protection Mechanism > Migrate to VADP.The vSphere Web Client shows a task for the virtual machine to disable the Lightweight Delta (LWD) filter.Select the dropdown More Actions > Protection Mechanism > Migrate to Transparent Snapshots.The vSphere Web Client shows a task for the virtual machine to enable the LWD filter.Run a backup or monitor the scheduled backup execution. Method 2 Remove and add the affected VM from the Protection Policy. Open a browser session to PowerProtect Data Manager and authenticate with an admin account.Browse to Infrastructure > Assets > Virtual Machines.Select the affected virtual machine.Select the dropdown More Actions > Remove from Policy.The vSphere Web Client shows a task for the virtual machine to disable the Lightweight Delta (LWD) filter.Select the dropdown More Actions > Add to Policy, and follow the wizard to add to the Protection Policy.The vSphere Web Client shows a task for the virtual machine to enable the LWD filter.Run a backup or monitor the scheduled backup execution. Method 3 The PowerProtect Data Manager vmconfig command-line utility can be used to change the SDM feature setting for the affected VM. Log in to the PowerProtect Data Manager Appliance using SSH with the admin account: ssh admin@my-ppdm-name Change to the root user: sudo su Load the environment variables: source /opt/emc/vproxy/unit/vproxy.env Change to the vProxy bin directory: cd /opt/emc/vproxy/bin/ Construct the vmconfig command by replacing the information in the brackets: Check the VM status: ./vmconfig -v -u '' -p '' -l dns -k -c info Disable SDM on the VM: ./vmconfig -v -u '' -p '' -l dns -k -c disable-sdm Enable SDM on the VM: ./vmconfig -v -u '' -p '' -l dns -k -c enable-sdm Run a PowerProtect Data Manager TSDM backup to confirm the issue is resolved.

Support Cases