Symptoms

The PowerProtect Data Manager Transparent Snapshot Data Mover (TSDM) solution is used to protect the vSphere environment. The virtual machine (VM) Protection Policy is encountering intermittent backup failures during the scheduled execution. The PowerProtect Data Manager User Interface (UI) shows critical message ID ABV0016: The full message text reads: ABV0016: VM Direct engine 'localhost' is unable to back up the virtual machine 'my-vm-name' on vCenter 'my-vcernter-name' because of a vCenter API issue. The VM Direct engine cannot complete the backup of this virtual machine because of a vCenter API operation did not complete successfully. To troubleshoot this issue: 1) Verify that the vCenter can be reached. 2) If the vCenter is busy, wait for some activities to complete and then retry the operation. 3) Verify that the virtual machine exists on the vCenter. Read Less Unable to create LWD snapshot. vCenter task 'task-#####' failed: VimFault: dp.vpx.fault.DpdThrottleLimitExceeded.summary' The backup session log shows: YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: SDM Full Sync: Closed file 'VM_Image_Daily_NON-SQL-uklpdtsbk001-f4723/PLCTLP-2832693c-377a-4f7e-952f-ffc1fdade2b1/Backups/c57b4143-f4c3-4760-86f5-7448429205ae/501efcd4-1f7e-5c9d-b98c-43633d01aa4c/lastSdmDiskBackupPath.json'. YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: SDM Full Sync: Checking if the last path exists which was used as a baseline for each disk ... YYYY-MM-DD HH:MM:SS INFO: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: SDM Full Sync: Full sync is not needed, skipping. YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: vCenter SOAP call 'DpSnapshot' was accepted. YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: The vCenter task failed: dp.vpx.fault.DpdThrottleLimitExceeded.summary YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: The vCenter task for 'Create SDM Snapshot' completed with state 'error'. YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: Unable to create SDM snapshot. YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: Retrying in 500ms... ... YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: vCenter SOAP call 'DpSnapshot' was accepted. YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: The vCenter task failed: dp.vpx.fault.DpdThrottleLimitExceeded.summary YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: The vCenter task for 'Create SDM Snapshot' completed with state 'error'. YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: Unable to create SDM snapshot. YYYY-MM-DD HH:MM:SS ERROR: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Create LWD Snapshot: Unsuccessful after 98 attempts over 1m0.158271936s. YYYY-MM-DD HH:MM:SS TRACE: [a30a8b8946958a4e;8d8911578980c3ff] SDM Data Mover: Snapshot failure can't be remedied by a Full sync. The vSphere web client shows at least one ESXi host with continuous LWD-based errors during the scheduled execution.The vCenter Support bundle /var/log/vmware/vsan-health/vsanvcmgmtd-*.log output shows the ESXi host is being identified as a legacy DPD host: YYYY-MM-DD HH:MM:SS warning vsanvcmgmtd[11593] [vSAN@6876 sub=dps] [opId=0849c23c] Found legacy DPD on host: host-29. Applying legacy throttle limit: '10', instead of regular throttle limit: '20'

Cause

The DPD service has dependencies with the vCenter vSAN Health Service, even when vSAN storage is not configured in the environment. The vCenter vSAN Health Service provides the ESXi version details to determine the number of concurrent SDM sessions supported by the ESXi host. The number of concurrent SDM backup sessions per ESXi host is 20 with vSphere version 7.0 U3d (ESXi and vCenter). The vSphere versions prior to vSphere 7.0 U3d (legacy versions) could only perform 10 concurrent SDM backup sessions. If the vCenter vSAN Health Service cache has outdated version information or a null entry, it categorizes the ESXi host as legacy. The default PowerProtect Data Manager throttle sends 18 SDM backup sessions per ESXi host. The ESXi host processes the initial 10 SDM backups, but the vCenter would fail any subsequent SDM backups indicating the DPD limit was exceeded.

Resolution

There are enhancements in vSphere 8.0 U1 that will automatically refresh the vSAN Health service cache in certain conditions. It is recommended to upgrade to vSphere 8.0 U1 to take advantage of the enhancements and prevent certain vSAN Health service cache symptoms. VMware Workaround:Ensure the vCenter server and all the ESXi hosts in the cluster are version 7.0 U3d or newer. To clear the cache, the vCenter Server Appliance (VCSA) vSAN Health service may be restarted over an SSH session: service-control --restart vsan-health If required, contact VMware Support for further insight or assistance with the vSAN Health service.

Support Cases