Symptoms

PPDM VMware VM backups are failing intermittently. Multiple VM failures are seen. The following errors are seen in the log BackupVmSessions-.log: YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleaning up after unexpected restart of session. YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleanup backup of "vm-xxxxx", "vmname1"... YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleaning up VDDK usage related to vCenter "vcsa01.mydomain.local"... YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: VMs still needing cleanup: 0. YYYY-MM-DDThh:mm:ss.xyzZ ERROR: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: The backup had terminated unexpectedly. YYYY-MM-DDThh:mm:ss.xyzZ TRACE: [9ebdacd6b621654a;a71a507c047110b8] Closed task engine's updateChan. . . YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Error": { YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Text": "The backup had terminated unexpectedly.", YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Code": 500, This failure is due to 'vbackupd' on the External Direct Engine (aka Proxy) restarting. The restarting occurred because of a VDDK API call timing out (see the following log extract).BackupJobSessions-.log YYYY-MM-DDThh:mm:ss.xyzZ TRACE: [9059979fb50f93a7;a9d5325355695f7f] Opening source VMDK '[DATASTORE01] vmname1/vmname1.vmdk' ... YYYY-MM-DDThh:mm:ss.xyzZ ERROR: [9059979fb50f93a7;a9d5325355695f7f] VDDK config call timed out after 1h0m0s, exiting 'vbackupd', PID=13261.

Cause

The ESX host is unable to handle the large number of VDDK calls being made to it. By default PPDM opens a MAX of 48 NDB sessions to the ESX host.

Resolution

Perform the following Tuning of the MAX_NBD_BACKUP_SESSIONS setting on the PPDM server to reduce the NBD load on the ESX host:1. Ensure no backups or restore running on the PPDM server.2. SSH to PPDM server as 'admin'3. su to root user4. Using VI or VIM, edit the file /opt/emc/vmdirect/unit/vmdirect.env. Change the value of the following attribute. It should be 25 instead of the default 48. Save the file.MAX_NBD_BACKUP_SESSIONS=48 ==> This is the default value. Change this to 25.5. Restart VISD with 'systemctl restart visd'.Also implement the retries in Workflow as below: https://dl.dell.com/content/manual44353552-powerprotect-data-manager-19-13-administration-and-user-guide.pdf?language=en-us#page=140

Support Cases