Symptoms

Avamar Garbage collection (GC) fails with MSG_ERR_DDR_ERROR. The following errors are seen in the Data Domain (DD) maintenance logs (ddrmaint.log): grep -i "Failed to read" /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log Aug 20 08:27:52 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/f0d09b747587b71c34f90b73a84b3a7866cb289c/1D3F60FB8F8404C/ddr_files.xml' Aug 20 08:27:52 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/f7f529c2372cdfb63eb9ca80ca70bbfc111c6680/1D3F608866DFF24/ddr_files.xml' Aug 20 08:27:53 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/b2d365bfe514e3dd8fd039cdeed24d5ce81b74ba/1D3F60961EEEDB0/ddr_files.xml' Aug 20 08:27:53 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/db5d9e3227847903d6a59ff0f395f0766c266bf5/1D3F6065268E920/ddr_files.xml' Aug 20 08:27:55 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/c97f81c65d7aa401076b3731e7d5b67a530e3da8/1D3F60F7EA6EB0A/ddr_files.xml' Aug 20 08:27:56 ddrmaint.bin[13345]: Error: gc-finish::validate_cloud_copy: Failed to read 'cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2/ddr_files.xml' grep -i "err.*maintain_cloud_copy: Cloud copy" /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log Aug 20 14:00:11 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/f0d09b747587b71c34f90b73a84b3a7866cb289c/1D3F60FB8F8404C' is corrupt Aug 20 14:00:12 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/f7f529c2372cdfb63eb9ca80ca70bbfc111c6680/1D3F608866DFF24' is corrupt Aug 20 14:00:12 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/b2d365bfe514e3dd8fd039cdeed24d5ce81b74ba/1D3F60961EEEDB0' is corrupt Aug 20 14:00:12 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/db5d9e3227847903d6a59ff0f395f0766c266bf5/1D3F6065268E920' is corrupt Aug 20 14:00:14 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/c97f81c65d7aa401076b3731e7d5b67a530e3da8/1D3F60F7EA6EB0A' is corrupt Aug 20 14:00:15 ddrmaint.bin[11789]: Error: gc-finish::maintain_cloud_copy: Cloud copy 'cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2' is corrupt Note: root privilege is required to review the ddrmaint.log. The logs also contain information about a recall failure: grep -i "recall-backup.*" ddrmaint.log* Example and sample output: grep -i "recall-backup.*9f30e9cd4833708ec979338a1d7ea8050d1eaf30" ddrmaint.log* ddrmaint.log.3:Aug 13 02:25:53 ddrmaint.bin[28806]: Info: recall-backup cmdline: '/usr/local/avamar/bin/ddrmaint.bin --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc --bindir=/usr/local/avamar/bin --sysdir=/usr/local/avamar/etc/client --vardir=/usr/local/avamar/var/client recall-backup --flagfile=/usr/local/avamar/etc/client/usersettings.cfg --server=avamartest --vardir=/usr/local/avamar/var --bindir=/usr/local/avamar/bin --id=root --password=**************** --dpnid=1522437690 --account=/clients/testabc --client=9f30e9cd4833708ec979338a1d7ea8050d1eaf30 --backups=1D3F6089358D0E2:1 --logfile=/usr/local/avamar/var/client/MOD-1533734850483-1051-Tiering_tmpdir/MOD-1533734850483-1051-Tiering-#154-ddrmaint.log' ddrmaint.log.3:Aug 13 02:27:45 ddrmaint.bin[28806]: Error: recall-backup::recall_backup - failed to clone cur/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2 to cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2, error code: 5040 ddrmaint.log.3:Aug 13 02:27:45 ddrmaint.bin[28806]: Error: recall-backup::recall_backup - failed to clone backup from cur/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2 to cloud-copy/9f30e9cd4833708ec979338a1d7ea8050d1eaf30/1D3F6089358D0E2

Cause

Avamar garbage collection may fail with MSG_ERR_DDR_ERROR after a Cloud recall failure caused due to DDFS panic. If the Data Domain file system panicked while files were being recalled from the cloud tier, the ongoing cloud recall process leaves some partially recalled backup files in the "/data01/col1/avamar-xxxxxx/cloud-copy/<CID>/<BID>" directory. Avamar Garbage collection (GC) then fails with MSG_ERR_DDR_ERROR. Garbage collection fails to read ddr_files.xml for backups in the cloud-copy directory on Data Domain. It is possible that the ddr.xml file is present in the cloud-copy directory under the <BID> directory however when the Garbage collection reads the ddr.xml file, it finds that all required backup files are not present which may cause the GC to fail with this error. This maybe because the ddr.xml is missing under directory "/data01/col1/avamar-xxxxxx/cloud-copy/<CID>/<BID>" or any of the backup files are missing.

Resolution

Create a Service Request, referencing this article.

Support Cases