Symptoms

The NetWorker VMware Protection integration is configured with the vProxy Appliance. The backups intermittently fail indicating that it is unable to open the previous backup on the Data Domain.Failed backup: ... YYYY/MM/DD HH:MM:SS NOTICE: [@(#) Build number: 85] "BackupPath": "[MTREE1_NAME]/[DEVICE_NAME]/[MTREE1_NAME]/[DEVICE_NAME]14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56", YYYY/MM/DD HH:MM:SS NOTICE: [@(#) Build number: 85] "PreviousBackupPath": "/[MTREE2_NAME]/[DEVICE_NAME]/52/00/c2b12b8b-00000006-9f9924fb-5b9924fb-12625000-0e1bbb56", YYYY/MM/DD HH:MM:SS NOTICE: [@(#) Build number: 85] "CurrentBackupPath": "/[MTREE1_NAME]/[DEVICE_NAME]/active/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56", ... YYYY/MM/DD HH:MM:SS TRACE: [@(#) Build number: 85] Getting previous backup descriptor from "/[MTREE2_NAME]/[DEVICE_NAME]/52/00/c2b12b8b-00000006-9f9924fb-5b9924fb-12625000-0e1bbb56" ... YYYY/MM/DD HH:MM:SS TRACE: [@(#) Build number: 85] Opening Backup Descriptor file "/[MTREE2_NAME]/[DEVICE_NAME]/52/00/c2b12b8b-00000006-9f9924fb-5b9924fb-12625000-0e1bbb56/BackupDescriptor.json" ... YYYY/MM/DD HH:MM:SS ERROR: [@(#) Build number: 85] [ 8625] [140323406313216] MM DD HH:MM:SS YYYY YYYY/MM/DD HH:MM:SS ERROR: [@(#) Build number: 85] ddp_open_file() failed for File: //[MTREE2_NAME]/[DEVICE_NAME]/52/00/c2b12b8b-00000006-9f9924fb-5b9924fb-12625000-0e1bbb56/BackupDescriptor.json, Err: 5034-nfs lookup failed (nfs: Permission denied) YYYY/MM/DD HH:MM:SS ERROR: [@(#) Build number: 85] Failed to open Backup Descriptor file "/[MTREE2_NAME]/[DEVICE_NAME]/52/00/c2b12b8b-00000006-9f9924fb-5b9924fb-12625000-0e1bbb56/BackupDescriptor.json". YYYY/MM/DD HH:MM:SS ERROR: [@(#) Build number: 85] Unable to open previous backup. ... The NetWorker Server daemon log shows: ... MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrd NSR info [VM_NAME]:vm:520b7dfe-356e-afad-0db4-5763aaa8d0e8:[VCENTER_NAME] saving to pool '[DATADOMAIN_POOL]' ([DATADOMAIN_POOL].003) ... MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrmmd SYSTEM critical Cannot determine file size for save set ID 1251623055: Unable to stat save set file '/[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56': Retrieving statistics for '/[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56' failed [5004] ([23131] [140678961694496] MM DD HH:MM:SS YYYY ddp_stat() failed, Path //[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56, Err: 5004-nfs lookup failed (nfs: No such file or directory) ). MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrmmd SYSTEM critical Cannot stat active file /[MTREE1_NAME]/[DEVICE_NAME]/active/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56: Unable to stat save set file '/[MTREE1_NAME]/[DEVICE_NAME]/active/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56': Retrieving statistics for '/[MTREE1_NAME]/[DEVICE_NAME]/active/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56' failed [5004] ([23131] [140678961694496] MM DD HH:MM:SS YYYY ddp_stat() failed, Path //[MTREE1_NAME]/[DEVICE_NAME]/active/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56, Err: 5004-nfs lookup failed (nfs: No such file or directory) ). MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrmmd NSR warning Removing ssid 1251623055 from disk volume [DATADOMAIN_NAME]_[DEVICE_NAME] and MMDB due to writing related failures. MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrmmd SYSTEM critical Unable to remove SSID file '/[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56' on device '[DATADOMAIN_NAME]_[DEVICE_NAME]': Unable to stat save set file '/[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56': Retrieving statistics for '/[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56' failed [5004] ([23131] [140678961694496] MM DD HH:MM:SS YYYY ddp_stat() failed, Path //[MTREE1_NAME]/[DEVICE_NAME]/14/38/05d6154a-00000006-4a9a408f-5b9a408f-17b75000-0e1bbb56, Err: 5004-nfs lookup failed (nfs: No such file or directory) ). MM/DD/YYYY HH:MM:SS THREAD [NSR_NAME] nsrd NSR info [VM_NAME]:vm:520b7dfe-356e-afad-0db4-5763aaa8d0e8:[VCENTER_NAME] done saving to pool '[DATADOMAIN_POOL]' ([DATADOMAIN_POOL].003) ... NetWorker: How to use nsr_render_log

Cause

The cause is not known currently. In one instance the target backup pool contained devices from different mtree's on the same Data Domain. When the backup workflow was using the "PreviousBackupPath" and the "CurrentBackupPath" devices on different mtree's the backup would fail. If the backup workflow was using the "PreviousBackupPath" and the "CurrentBackupPath" devices on the same mtree, the backups would complete successfully. The target backup pool must contain devices from the same Data Domain, but it is supported to have the devices in different mtree's.

Resolution

Ensure that the Data Domain devices are properly configured in the environment. If required, create a new target backup pool with devices under the same Data Domain mtree.This is not reproducible in other environments with backup pools containing devices from multiple mtree's. As the different mtree devices can save to their "PreviousBackupPath", this may be a lower-level Data Domain issue accessing the file system of the different mtree's. If required, contact Dell EMC Data Domain Technical Support for further investigation of the Data Domain file system.

Support Cases