Symptoms

Scenario 1Data Domain showing red in AUI and or user interface due to certificate issues, which may also be causing backup and or replication failures. Scenario 2Data Domain is showing red in the AUI and or user interface due to incorrect SNMP configuration.Scenario 3Data Domain is showing red in AUI and or user interface due to missing and or incorrect ddr_key.Scenario 4Expired certificates.Scenario 5The entry key, "hfsaddr" in mcserver.xml is configured as ip instead of hostname, while the subject of imported-ca is Avamar hostname.

Cause

Certificate, SNMP, or Public Key misconfiguration.

Resolution

Data Domain showing red in AUI and or user interface due to certificate issues, which may also be causing backup and or replication failures. Goav Tool Automation The detailed scenarios in this article can be followed manually, or the Goav command line (CLI) tool can be used to automatically detect issues and resolve them.See the knowledge base article for more details on using Goav to resolve the issues described in KB article 000215679, Avamar: Information About Goav dd check-ssl Feature. How to Fix Data Domain Certificate issues in Avamar using the GoAV Tool dd check-ssl Feature Duration: 00:01:38 (hh:mm:ss)When available, closed caption (subtitles) language settings can be chosen using the CC icon on this video player. You can also view this video on YouTube. Scenario 1The procedure for scenario 1 is only relevant when session security is enabled.Check if session security is enabled as root user: enable_secure_config.sh --showconfig Current Session Security Settings ---------------------------------- "encrypt_server_authenticate" ="false" "secure_agent_feature_on" ="false" "session_ticket_feature_on" ="false" "secure_agents_mode" ="unsecure_only" "secure_st_mode" ="unsecure_only" "secure_dd_feature_on" ="false" "verifypeer" ="no" Client and Server Communication set to Default (Workflow Re-Run) mode with No Authentication. Client Agent and Management Server Communication set to unsecure_only mode. Secure Data Domain Feature is Disabled. The output above shows session security disabled.Anything other than the output shown above indicates that session security is enabled.Example: enable_secure_config.sh --showconfig Current Session Security Settings ---------------------------------- "encrypt_server_authenticate" ="true" "secure_agent_feature_on" ="true" "session_ticket_feature_on" ="true" "secure_agents_mode" ="secure_only" "secure_st_mode" ="secure_only" "secure_dd_feature_on" ="true" "verifypeer" ="yes" Client and Server Communication set to Authenticated mode with Two-Way/Dual Authentication. Client Agent and Management Server Communication set to secure_only mode. Secure Data Domain Feature is Enabled. Symptoms:DDR result code: 5049, desc: File not foundDDR result code: 5341, desc: SSL library error "failed to import host or ca certificate automatically"DDR result code: 5008, desc: Invalid argument Cause:All these result codes on failure to backup to data domain when session security is enabled relate to certificate issues. Resolution:Here are the steps to ensure that certificate imports are automatic and correct. Verify that there is a system passphrase set on Data Domain before proceeding to check certificates. On Data Domain Enterprise Manager User Interface, go to Administration > Access > Administrator Access. The button labeled "CHANGE PASSPHRASE" shows that the system passphrase is set.1. On Data Domain, check the current certificates. ddboost51@fudge# adminaccess certificate show Subject Type Application Valid From Valid Until Fingerprint ----------------------------------- ------------- ----------- ------------------------ ------------------------ ------------------------------------------------------------ fudge_dd.net host https Sun Nov 5 12:16:05 2017 Wed Oct 28 18:16:05 2048 5B:58:0A:83:C4:3E:06:91:51:C7:87:F2:45:82:48:95:99:E4:48:B5 fudge_dd.net ca trusted-ca Tue Jun 26 16:36:14 2012 Fri Jun 19 16:36:14 2043 44:DD:C1:61:14:5B:54:BE:41:1F:BF:40:9C:2E:6F:A3:02:2F:18:9A fudge_dd.net imported-host ddboost Wed Jan 19 12:22:07 2022 Mon Jan 18 12:22:07 2027 63:50:81:4B:B3:9B:2A:29:38:57:62:A8:46:2E:A9:D7:EF:32:12:F5 fudge_av.com imported-ca ddboost Thu Jan 6 10:16:07 2022 Tue Jan 5 10:16:07 2027 FC:57:B7:1B:5B:F0:FA:79:54:B0:B4:52:1B:D8:15:2F:CE:9D:F5:10 ----------------------------------- ------------- ----------- ------------------------ ------------------------ ------------------------------------------------------------ 2. Delete any imported certificates for the Avamar that is experiencing backup failures, for example: fudge_av.com which is the Avamar listed in the output of command "adminaccess certificate show." ddboost51@fudge# adminaccess certificate delete subject fudge_av.com 3. Delete the imported-host ddboost certificate. ddboost51@fudge# adminaccess certificate delete imported-host application ddboost 4. Check current certificates after the deletion. ddboost51@fudge# adminaccess certificate show Subject Type Application Valid From Valid Until Fingerprint ----------------------------------- ------------- ----------- ------------------------ ------------------------ ------------------------------------------------------------ fudge_dd.net host https Sun Nov 5 12:16:05 2017 Wed Oct 28 18:16:05 2048 5B:58:0A:83:C4:3E:06:91:51:C7:87:F2:45:82:48:95:99:E4:48:B5 fudge_dd.net ca trusted-ca Tue Jun 26 16:36:14 2012 Fri Jun 19 16:36:14 2043 44:DD:C1:61:14:5B:54:BE:41:1F:BF:40:9C:2E:6F:A3:02:2F:18:9A 5. Check mcserver.xml parameters. On Avamar version 19.3 and below: admin@fudge_av.com:/usr/local/avamar/var/mc/server_data/prefs/>: grep -i manual mcserver.xml <entry key="ddr_security_feature_manual" value="false" /> On Avamar version 19.4: grep -i "manual|ddr_host" /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml admin@fudge_av.com:/usr/local/avamar/var/mc/server_data/prefs/>: grep -i "manual|ddr_host" mcserver.xml <entry key="ddr_host_cert_auto_refresh" value="false" /> <entry key="ddr_security_feature_manual" value="false" /> 6. Ensure that the manual security feature is set to false. This allows the certificates to automatically import to the Data Domain. On Avamar 19.3 and below, if it is set to true then set it to false and restart MCS. <entry key="ddr_security_feature_manual" value="false" /> On Avamar 19.4 and later, you can set both flags to false and restart MCS. <entry key="ddr_host_cert_auto_refresh" value="false" /> <entry key="ddr_security_feature_manual" value="false" /> 7. Restart MCS. mcserver.sh --stop mcserver.sh --start 8. On data domain, restart ddboost. ddboost disable ddboost enable 9. Open the Avamar user interface and or AUI, and update and or edit the Data Domain System.Open the Data Doman server in the Avamar Administrator.In Avamar MCGUI, go to Server > Server Management, select the DD server, click on Edit Data Domain System icon and click on OK in the display window.a. In Avamar Administrator, click the Server launcher button. The Server window appears.b. Click the Server Management tab.c. Select the Data Domain system to edit.d. Select Actions > Edit Data Domain System. The Edit Data Domain System dialog box appears.e. Click OK.No changes are required for the Data Domain configuration. 10. After the edit is complete the certificates should be automatically imported to the Data Domain. ddboost51@fudge# adminaccess certificate show Subject Type Application Valid From Valid Until Fingerprint ------------------------------- ------------- ----------- ------------------------ ------------------------ ------------------------------------------------------------ fudge_dd.net host https Sun Nov 5 12:16:05 2017 Wed Oct 28 18:16:05 2048 5B:58:0A:83:C4:3E:06:91:51:C7:87:F2:45:82:48:95:99:E4:48:B5 fudge_dd.net ca trusted-ca Tue Jun 26 16:36:14 2012 Fri Jun 19 16:36:14 2043 44:DD:C1:61:14:5B:54:BE:41:1F:BF:40:9C:2E:6F:A3:02:2F:18:9A fudge_dd.net imported-host ddboost Fri Feb 25 13:29:36 2022 Wed Feb 24 13:29:36 2027 4F:B3:68:1C:F7:EB:25:F5:F1:81:F1:38:3B:B7:06:6B:DD:04:C1:33 fudge_av.com imported-ca ddboost Mon Feb 7 13:30:20 2022 Sat Feb 6 13:30:20 2027 FC:57:B7:1B:5B:F0:FA:79:54:B0:B4:52:1B:D8:15:2F:CE:9D:F5:10 ------------------------------- ------------- ----------- ------------------------ ------------------------ ------------------------------------------------------------ 11. Remember to resume the backup scheduler on Avamar if needed. dpnctl start sched If this procedure fails to import the certificates , please check that Avamar and Data Domain times are in sync, KB article 000197106 otherwise the Avamar certificate will not be valid yet.Scenario 2Data Domain is showing red in the AUI and or user interface due to incorrect SNMP configuration. Symptoms:In Java user interface and or AUI, DD showing red on the main screen. Cause:Incorrect DD SNMP Config can also cause the DD to show red or 0s in the user interface and or AUI. Resolution:Verifying and or Correcting DD SNMP Config The easiest way to verify and or correct DD SNMP version 2 is using the DD web interface. https://<data_domain_fqdn> Navigate the interface to Administration > Settings > SNMP > SNMP V2C Configuration. 1. Create a read-only community string or use an existing one. 2. Create a trap host which is the Avamar hostname, port 163, and select the community string you want to use. 3. Go to the Avamar Java user interface or AUI, and edit the Data Domain system, Select SNMP tab, and update the SNMP community string that you configured for your trap host. 4. You may have to restart the "mcddrnsmp" service on Avamar, as root: mcddrsnmp restart Related Lightning Knowledge Based Articles for SNMP configuration:KB article 000063895, Data Domain: Common SNMP configuration and Issues causing Monitoring Services disabled in Integrated Backup Software or DPAScenario 3Data Domain is showing red in AUI and or user interface due to missing and or incorrect ddr_key. When an Avamar system stores backups on a Data Domain system, the Avamar Management Console Server (MCS) issues commands to the Data Domain system using the SSH protocol. This protocol provides a secure communication channel for remote command execution. To permit remote command execution using SSH, Data Domain systems provide an SSH interface named DDSSH. The DDSSH interface requires authentication of the Avamar system. Authentication is accomplished by creating SSH private and public keys on the Avamar system and sharing the public key with the Data Domain system.1. On Avamar, open a command shell, log in to Avamar and load the keys. ssh-agent bash ssh-add ~admin/.ssh/admin_key 2. Check that the ddr_key and ddr_key.pub are already in the folder /home/admin/.ssh/: ls -lh /home/admin/.ssh/ddr* 3. Open the ddr_key.pub with cat and copy its content. It is useful to paste on Data Domain later. cat /home/admin/.ssh/ddr_key.pub 4. Copy the entire content of the file s it is required later. It looks like this: ssh-rsa AAAAB3NzaC1yc2EAAAOSDFkNBGH177bvYPHrAqW5nXEw6uZwV7q0k9SLHgirfv2AztJcCuJIW8LKN0MBTYArGhRJRWE9etR3hH[...]0NxtMIZyhIWKas+PJ0J/AgJhl admin@avamarhostname 5. Log in to the Data Domain system by typing: ssh <ddboost>@<DataDomainHostname> 6. Check the ssh-keys adminaccess show ssh-keys 7. Use the Data Domain command adminaccess add ssh-keys to open the keystore on the Data Domain system: adminaccess add ssh-keys user <ddboost> Where <ddboost> is the username assigned to the Avamar system on the Data Domain system. The utility prompts for the key: ddboost@datadomain# adminaccess add ssh-keys user ddboost Enter the key and then press Control-D, or press Control-C to cancel.8. Paste the SSH public key of the Avamar system (ddr_key.pub) at this prompt9. Complete the entry of the key by pressing Ctrl+D to save it. The utility adds the public key to the keystore on the Data Domain system.10. Log out of the Data Domain system. exit 11. Back to Avamar, load the ddr keys. ssh-agent bash ssh-add ~/.ssh/ddr_key 12. Test that you can log in to the Data Domain system without providing a password by typing: ssh <ddboost>@<DataDomainHostname> admin@avamar:~/#: ssh ddboost@DataDomainHostname EMC Data Domain Virtual Edition Last login: Tue Dec 3 01:17:07 PST 2019 from 10.x.x.x on pts/1 Welcome to Data Domain OS 6.2.0.10-615548 ----------------------------------------- ddboost@DataDomainHostname# Scenario 4The Avamar server/gsan certificates have expired, causing backups to fail.The Data Domain imported-host ddboost certificate has expired, causing backups to fail. If the Avamar server/gsan certs have expired, you must regenerate ALL certificates using the session security AVP. We select ALL certificates because the avamar_keystore must get new root keys in order to make new server/gsan certificates from those keys. Use the following KB article to download, and install the session security avp to regenerate all certificates.KB article 000067229 Avamar-IDPA: Backups or replications fail with certificate error. After regenerating the certificates, the Data Domain must get the new imported-ca ddboost (Avamar chain.pem).Scenario 5Contact Dell Support for assistance and mention this article ID.

Support Cases