Loading...
Loading...
PPDM VMware VM backups are failing intermittently. Multiple VM failures are seen. The following errors are seen in the log BackupVmSessions-<xxxxx>.log : YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleaning up after unexpected restart of session. YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleanup backup of "vm-xxxxx", "vmname1"... YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: Cleaning up VDDK usage related to vCenter "vcsa01.mydomain.local"... YYYY-MM-DDThh:mm:ss.xyzZ INFO: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: VMs still needing cleanup: 0. YYYY-MM-DDThh:mm:ss.xyzZ ERROR: [9ebdacd6b621654a;a71a507c047110b8] Cleanup Backup: The backup had terminated unexpectedly. YYYY-MM-DDThh:mm:ss.xyzZ TRACE: [9ebdacd6b621654a;a71a507c047110b8] Closed task engine's updateChan. . . YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Error": { YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Text": "The backup had terminated unexpectedly.", YYYY-MM-DDThh:mm:ss.xyzZ NOTICE: [8edf1170cc271f34;8ee0488b4d2dcbaf] "Code": 500, This failure is due to ' vbackupd ' on the External Direct Engine (aka Proxy) restarting. The restarting occurred because of a VDDK API call timing out (see the following log extract). BackupJobSessions-<xxxxxxx>.log YYYY-MM-DDThh:mm:ss.xyzZ TRACE: [9059979fb50f93a7;a9d5325355695f7f] Opening source VMDK '[DATASTORE01] vmname1/vmname1.vmdk' ... YYYY-MM-DDThh:mm:ss.xyzZ ERROR: [9059979fb50f93a7;a9d5325355695f7f] VDDK config call timed out after 1h0m0s, exiting 'vbackupd', PID=13261.
The ESX host is unable to handle the large number of VDDK calls being made to it. By default PPDM opens a MAX of 48 NDB sessions to the ESX host.
Perform the following Tuning of the MAX_NBD_BACKUP_SESSIONS setting on the PPDM server to reduce the NBD load on the ESX host: 1. Ensure no backups or restore running on the PPDM server. 2. SSH to PPDM server as 'admin' 3. su to root user 4. Using VI or VIM, edit the file /opt/emc/vmdirect/unit/vmdirect.env. Change the value of the following attribute. It should be 25 instead of the default 48. Save the file. MAX_NBD_BACKUP_SESSIONS= 48 ==> This is the default value. Change this to 25. 5. Restart VISD with ' systemctl restart visd '. Also implement the retries in Workflow as below: https://dl.dell.com/content/manual44353552-powerprotect-data-manager-19-13-administration-and-user-guide.pdf?language=en-us#page=140
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.