Symptoms

Avamar with Data Domain integrated backups fail. When the avtar logs are consulted the error appears to be an invalid token: 2017-02-21 11:35:50 avtar Info : - Client connecting to the Avamar Server using authentication, client connecting to the Data Domain system using two-way authentication 2017-02-21 11:35:51 avtar Info : - Resolved Data Domain Server name "testdd.emc.com" to the IP address "10.241.170.73" 2017-02-21 11:35:51 avtar Info : - Connecting to Data Domain Server name "testdd.emc.com" with token:5ba93c9db0cff93f52b521d7420e43f6eda2784f 2017-02-21 11:35:51 avtar Error : Using invalid token:5ba93c9db0cff93f52b521d7420e43f6eda2784f 2017-02-21 11:35:51 avtar Error : Data Domain server "testdd.emc.com" open failed DDR result code: 4904, desc: Invalid API argument. 2017-02-21 11:35:51 avtar Error : Problem logging into the DDR server:'', only GSAN communication was enabled. 2017-02-21 11:35:51 avtar FATAL : Backup is incomplete because file "/ddr_files.xml" is missing Consultation of /usr/local/avamar/var/ddrmaintlogs/ddrmaint.log shows that the token request failed: Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Info: request-token:open_ddr:service handle:1 index:1 server:testdd.emc.com user:ddboost duration=1800 expires=2017-02-21 13:05:23 Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Error: request-token::body - Failed to get token. Use token=1:00. Error:5008 Reason:invalid argument Feb 21 11:35:23 av-un-ge06-adm0 ddrmaint.bin[47285]: Error: Datadomain request token operation failed. Finally review of the /ddr/var/log/debug/ddfs.info log on the Data Domain reveals that the token encryption failed due to invalid key magic:1. Login to the Data Domain.2. Review the ddfs.info log by using the following command: log view debug/ddfs.info Alternatively if it is preferred KB 323235 can be used to escape to bash mode to navigate directly to ddfs.info. 02/21 11:35:23.543 (tid 0x7f53bd82d990): ddboost-: testavamar.emc.com Local Time: Tue Feb 21 12:35:23 2017 02/21 11:35:23.553 (tid 0x7f53bd8078e0): nfsproc3_ost_get_token_3_svc: ost_build_token 02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_build_token() 02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_encrypt_string_with_key() This key has an invalid key magic 9df5c181 02/21 11:35:23.553 (tid 0x7f53bd8078e0): ost_build_token: Token encryption failed with error 5008 02/21 11:35:23.553 (tid 0x7f53bd8078e0): nfsproc3_ost_get_token_3_svc: ost_build_token failed [DDErrno = 5008 (ost_encrypt_string_with_key() Invalid User Key, Cannot be used to decrypt string)] If the invalid key magic messages are not present within the ddfs.info log please do not continue using this KB.

Cause

Resolution

1. Revoke token access from the ddboost user: ddboost user revoke token-access 2. Open the Data Doman server in the Avamar Administrator:- In Avamar MCGUI, go to Server > Server Management, select the DD server, click on "Edit Data Domain System icon" and click on OK in the pop-up window. a. In Avamar Administrator, click the Server launcher button. The Server window appears. b. Click the Server Management tab. c. Select the Data Domain system to edit. d. Select Actions > Edit Data Domain System. The Edit Data Domain System dialog box appears. e. Click OK. There is no need to make any changes to the Data Domain configuration. 3. Restart DDBoost on the Data Domain system: a. Log into the Data Domain System. b. Enter the following commands in the Data Domain CLI: ddboost disable ddboost enable 4. Test another backup to ensure this resolved the invalid token issue.

Support Cases