Symptoms

The following errors can be seen in the NDMP restore job log: 2018-12-06 15:03:04 avndmp Info : [avndmp_assist] Plugin exited with 'code 157: miscellaneous error'2018-12-06 15:03:04 avndmp Error : [avndmp_assist] Backup is invalid or not a NDMP backup.2018-12-06 15:03:04 avndmp Info : [avndmp_ctl_sup] Global avndmp_assist exit code is: 161 To Verify the ddboost plugin version on Avamar, run the following command: strings /usr/local/avamar/lib/libDDBoost.so | grep "[0-9]\.[0-9]\.[0-9]\.[0-9]" Or grep -i engine ddrmaint.log | tail -1 In DDFS log (/ddr/var/log/debug/ddfs.info) on Data Domain, the following errors are seen. Note: The NDMP accelerator node IP address can be searched in the ddfs.info logs on Data Domain to confirm the issue: 12/06 15:02:45.099 (tid 0x90a9b30): nfs_rpc_svc_idx0 accepted cfa 255 from 10.12.255.85:3991012/06 15:02:45.103 (tid 0x7ff35b151220): nfsproc3_ostmntsec_3_svc: connection failed permission (corrupted credentials) from host 10.12.255.85 - ost_decrypt_mnt_sec_request(): failed to finalize plain text (101077092, error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt) This confirms that DD was unable to decrypt the password provided for the NDMP Restore.

Cause

Avamar server 7.5.0 uses DDBoost library version 3.4.0.2. This issue exists on DDBoost library 3.4.0 and is fixed on 3.4.1.Avamar server version 7.5.1 has DDBoost library version 3.4.1.1 embedded in it. Starting from version 3.4 plugin, Data Domain does authentication using pre-shared-keys (PSKs) generated from the password. The client generates a PSK from the application supplied password, encrypts some authentication information, and sends it to DDR. The DDR will try to generate the same PSK (since it already knows the password for a user) and decrypt the contents the client sent and validate.With DDBoost ifgroup enabled, data domain falls into the reconnect path, which calls the PSK connection path and relies on the password crypt_hash stored in the nfs_conn structure. On DDBoost plugin 3.4.0.2 though, Data Domain does not generate the password crypt_hash again during re-connection. The PSK key is not the same on the client and DDR, hence the decryption failure.

Resolution

Workaround: Use one of the following methods to get around this problem. Method 1: Use username-password authentication. (Disable session security on the Avamar Server) Follow the information in KB article 484773: Avamar 7.3 and later: Backups or replications fails with certificate error to download the SessionSecurityConfiguration avp: Pre-checks before installing avp: Verify maintenance task, backup, restore, and replication is not running while installing SessionSecurityConfiguration avpVerify Avamar server has a current checkpoint Navigate to the Avamar Installation Manager (AVI) (http://<IP/fqdn of the avamar server/avi) page in a Web Browser and log in as root user: Under the Maintenance tab, select SessionSecurityConfiguration and click on Run: Select Disabled mode from the drop down menu under Security settings tab: Click on Continue. Follow the information in (Level 50 Internal) KB article 492934: Avamar: How to Disable SessionSecurityConfiguration Method 2: Upgrade the Avamar Client Catalogue to version 7.5.1-101 and then upgrade the NDMP Accelerator node to the same version (7.5.1).Method 3: Disable the DDBoost ifgroups on the Data domain. Note: Be aware that disabling ifgroups might directly impact the backup and restore performance. Log in to the Data Domain: Run the below command to view the existing ifgroups on the DD: ddboost ifgroup show config allConfirm which ifgroups are being utilized for backups. Make a note of the ifgroup name.Example: Group-name Status Interface Clients Replication----------- -------- --------- ------- -----------default disabled 0 1 0backupgroup enabled 2 0 0----------- -------- --------- ------- -----------Group-name Status Interfaces----------- ------- -----------backupgroup enabled 10.x.x.xbackupgroup enabled 192.x.x.x----------- ------- -----------Note: In the above example, the ifgroup named "backupgroup" is enabled and in use. Run the following command to disable the ifgroup in question: ddboost ifgroup disable

Support Cases