Symptoms

Password Out of Sync error message shows on the IDPA ACM Dashboard complaining about Avamar Server or AVProxy.Here is a Password Out of Sync example: Protection Software Proxy 'root' user out of sync. Please ensure 'root' password is same as 'admin' password

Cause

ACM monitors and reports all point-products which include its hardware and hypervisor platform health status. In order to do that, it keeps a copy of all current point-product login information in an encrypted password file. It constantly makes connections to those targets to check their health status. If it fails to log in to any of the monitoring target machines, it reports a password out of sync error in the ACM dashboard. Possible causes: Password is changed or is reset directly at the Avamar server or proxy end instead of from the ACM dashboard Network latency from ACM to Avamar Server or AVProxy to query its status. (temporary issue)

Resolution

If updating the password in ACM UI is unable to resolve the password out of sync issues, following these steps: Resolutions for Avamar server password out of sync: Scenario #1: Password changed from Avamar side instead of from ACM UI: Avamar password out of sync (one or more of the following Avamar user passwords have changed) Avamar OS root user passwordAvamar OS admin user passwordAvamar Server root user passwordAvamar Server MCUser user passwordAvamar Server repluser user passwordAvamar PostgreSQL database viewuser user passwordAvamar vProxy OS root user password (Appliance Internal proxy VM)Avamar vProxy OS admin user password (Appliance Internal proxy VM) Resolution: We strongly recommend keeping all Avamar Server and Avamar Proxy user passwords the same as IDPA common password, however if there is a requirement to keep them different, ensure the following rules are met: Password rules: If you change Avamar passwords from Avamar side instead of from ACM UI, IDPA requires that the Avamar OS admin and OS root accounts use the same password. The MCUser, repluser, viewuser, and Avamar server root user must share a same password which can be different than the OS admin and OS root passwords.In the above password list, the passwords highlighted in the same color MUST share a same password, and also all passwords have to meet the IDPA global password policy shown as below: The following procedure is to verify Avamar passwords: How to verify Avamar Server and Avamar Proxy machine OS root/admin passwords are the same? SSH login to Avamar server and Proxy as admin, then su to root user with the same password to see if the password can log in both admin and root account. How to verify the Avamar Server root, MCUser and repluser passwords? (Avamar Server root is different from Avamar Server OS root. OS root is a Linux OS level login user, and Avamar Server root is an Avamar application level user) SSH log in to the Avamar server as admin and run the following command four times with a different username: (replace with MCUser, repluser, viewuser, and root) # avmgr logn --id= --ap= 1 Request succeeded 7161 privilege level (enabled,create,read,backup,access,move,delete,maint,fullmanage,noticketrequired) 2 block type (directory) How to verify the Avamar Server mcdb viewuser password? SSH to the ACM machine and run: (if you run this command from the Avamar machine itself, it does not prompt for a password): # psql -U viewuser -h -p 5555 mcdb -c "\d" Password for user viewuser: The following procedure is to change Avamar passwords: Change Avamar Server-side passwords: Run the change-passwords command to change passwords. Here is an example to change all Avamar server passwords (in the real case, you can selectively change Avamar passwords that are set incorrectly). login as: admin Password: xxxxx admin@Avamar-svr:~/>: su - Password: xxxxx root@Avamar-svr:~/#: root@Avamar-svr:~/#: change-passwords [change-passwords version 2.1] Identity added: /root/.ssh/rootid (/root/.ssh/rootid) Identity added: /root/.ssh/rootid (/root/.ssh/rootid) Identity added: /root/.ssh/rootid-save (/root/.ssh/rootid-save) Do you wish to specify one or more additional SSH passphrase-less private keys that are authorized for root operations? Answer n(o) here unless there are known inconsistencies in ~root/.ssh/authorized_keys files among the various nodes. Note that the following keys will be used automatically (i.e., there is no need to re-specify them here): /root/.ssh/rootid /root/.ssh/rootid-save y(es), n(o), h(elp), q(uit/exit): no -------------------------------------------------------- The following is a test of OS root authorization with the currently loaded SSH key(s). If the authorization test fails, then you might be missing an appropriate private key, e.g., rootid or dpnid. -> In that event, re-run this program and, when prompted, specify as many SSH private key files as are necessary in order to complete root operations. Starting root authorization test with 600 second timeout... End of root authorization test. -------------------------------------------------------- Change OS (login) passwords? y(es), n(o), q(uit/exit): yes change-passwords: INFO: Each OS password will be changed locally without further prompting as soon as you have (twice) entered a valid password. -------------------------------------------------------- Change OS password for "admin"? y(es), n(o), q(uit/exit): yes Change password for user "admin". (Entering an empty (blank) line twice quits/exits.) > xxxxx Enter the same OS user password again. (Entering an empty (blank) line twice quits/exits.) > xxxxx BAD PASSWORD: it is too simplistic/systematic Backup lockbox file Backup keystore files Backup SSV files Flush backup Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00 Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup Updated with new value under name "admin". Backup lockbox file Backup keystore files Backup SSV files Flush backup Local backup dir: /usr/local/avamar/src/lockbox_backup/2023-06-26-22_00 Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup change-passwords: INFO: The password for OS user admin has been updated on _this_ host. change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases. -------------------------------------------------------- Change OS password for "root"? y(es), n(o), q(uit/exit): yes Change password for user "root". (Entering an empty (blank) line twice quits/exits.) > xxxxx Enter the same OS user password again. (Entering an empty (blank) line twice quits/exits.) > xxxxx BAD PASSWORD: it is too simplistic/systematic change-passwords: INFO: The password for OS user root has been updated on _this_ host. change-passwords: INFO: The password will not be reverted if you later decline to update passwords/passphrases. -------------------------------------------------------- Generate new SSH keys? y(es), n(o), h(elp), q(uit/exit): no -------------------------------------------------------- Change Avamar Server passwords? y(es), n(o), q(uit/exit): yes -------------------------------------------------------- Please enter the CURRENT server password for "root" (Entering an empty (blank) line twice quits/exits.) > xxxxx Checking Avamar Server root password (1200 second timeout)... Avamar Server current root password accepted. -------------------------------------------------------- Change Avamar Server password for "MCUser"? y(es), n(o), q(uit/exit): yes Please enter a new Avamar Server password for user "MCUser". (Entering an empty (blank) line twice quits/exits.) > xxxxx Enter the same Avamar Server password again. (Entering an empty (blank) line twice quits/exits.) > xxxxx Accepted Avamar Server password for "MCUser". -------------------------------------------------------- Change Avamar Server password for "root"? y(es), n(o), q(uit/exit): yes Please enter a new Avamar Server password for user "root". (Entering an empty (blank) line twice quits/exits.) > xxxxx Enter the same Avamar Server password again. (Entering an empty (blank) line twice quits/exits.) > xxxxx Accepted Avamar Server password for "root". -------------------------------------------------------- Change Avamar Server password for "repluser"? y(es), n(o), q(uit/exit): yes Please enter a new Avamar Server password for user "repluser". (Entering an empty (blank) line twice quits/exits.) > xxxxx Enter the same Avamar Server password again. (Entering an empty (blank) line twice quits/exits.) > xxxxx Accepted Avamar Server password for "repluser". -------------------------------------------------------- Change the viewuser password? y(es), n(o), h(elp), q(uit/exit): yes Checking Administrator Server status... Enter the NEW viewuser password. Enter ? or help for help. (Entering an empty (blank) line twice quits/exits.) > xxxxx For verification, re-enter the NEW viewuser password. Enter ? or help for help. (Entering an empty (blank) line twice quits/exits.) > xxxxx -------------------------------------------------------- Do you wish to proceed with your changes on the selected node? Answering y(es) will proceed to make changes. Answering n(o) or q(uit) will not proceed. y(es), n(o), q(uit/exit): yes Changing OS passwords... [Logging to /usr/local/avamar/var/change-passwords.log...] Done changing OS passwords... Changing Avamar Server passwords... Suspending maintenance cron jobs Checking Administrator Server status... Stopping Administrator Server... Changing the passwords for the local Avamar Server... The passwords for the local Avamar Server have been changed. Starting process of updating Administrator and Enterprise Manager configurations... Running script to update Administrator and Enterprise Manager configurations on node 0.s... [Logging to /usr/local/avamar/var/change-passwords.log...] Done with updating Administrator configuration on node 0.s... Starting process of updating client configurations... Running script to update client configuration on all+... [Logging to /usr/local/avamar/var/change-passwords.log...] Updating client configuration on node 0.0... Done updating client configuration on 0.0... Starting process of updating mccli configuration files... Running script to update mccli configuration files on node set "0.0"... [Logging to /usr/local/avamar/var/change-passwords.log...] Done with updating mccli configuration files on node 0.0... Checking Administrator Server status... Starting Administrator Server... Resuming maintenance cron jobs Starting process of updating viewuser password... Checking Administrator Server status... Stopping Administrator Server... Running script to update mcdb viewuser password on node 0.0... [Logging to /usr/local/avamar/var/change-passwords.log...] Done with updating mcdb viewuser password on node 0.0... Checking Administrator Server status... Starting Administrator Server... Stopping EMT subsystem Starting EMT subsystem -------------------------------------------------------- Done. NOTES: - If mccli (the Administrator command line interface) is used from any remote user accounts, then please update the password in each remote account's copy of the mccli preferences/configuration file, typically ~USER/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml. - Please be sure to resume schedules via the Administrator GUI or via 'dpnctl start sched'. #: dpnctl start sched Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key) dpnctl: INFO: Resuming backup scheduler... dpnctl: INFO: Backup scheduler resumed. dpnctl: INFO: No /usr/local/avamar/var/dpn_service_status exist. Change Avamar Proxy side passwords: Here is an example of changing both proxy admin and root passwords (login as root first, then change both admin and root passwords): login as: admin Password: xxxxx su - Password: xxxxx # passwd admin New password: xxxxx BAD PASSWORD: it is too simplistic/systematic BAD PASSWORD: is too simple Retype new password: xxxxx passwd: password updated successfully # passwd root New password: xxxxx BAD PASSWORD: it is too simplistic/systematic BAD PASSWORD: is too simple Retype new password: xxxxx passwd: password updated successfully Once all Avamar password rules are met, go to ACM UI, refresh the ACM UI browser page, then click the Out of Sync error message, it prompts you to enter new password for one or more users, update the password accordingly. (Sometimes it still shows a password out of sync error after you entered correct password, wait for a few minutes and refresh your web browser page again. Root cause shows in Scenario #2). Scenario #2 Password out of sync error due to network latency when ACM trying to query its point-products: This is a temporary issue and typically can be resolved if you refresh the ACM page after 1-2 minutes. This is a known issue, and the Dell engineering team is working on a fix in a future release. Scenario #3 Even though password is synchronized and works on Avamar, ACM shows password out of sync for AV, due to SSH failure or test connection failure to AV. This may be due to ACM failing to log in to AV due to SSH issues like recent changes made on AV sshconfig, cipher negotiation, and so on. Run a test SSH connection from ACM to Avamar server. If it fails, log in to Avamar server and restart SSH service: # service sshd restart If this does not help, gather the error message, troubleshooting steps you have performed and raise a ticket with Dell technical support for further assistance. Scenario #4 Avamar MCUser or viewuser may show out of sync when ACM is unable to perform MCSDK call to Avamar to validate those user passwords. This can happen if the ACM MCSDK call fails to Avamar due to various reasons. IDPA: ACM Reports Avamar Server Passwords Out of Sync Following Upgrade to 2.7.1 Upgrade or Application of Avamar 19.4 MCS Hotfix 333618 (Log in as Dell Support registered user is required to view article)IDPA: ACM UI shows error "Backup Server viewuser user password is out of sync. Update the latest password" on Backup Server Dashboard If the above provided scenarios and resolutions are unable to fix the issue, do the following: SSH Login to ACM as root, and stop and start ACM web application service: # service dataprotection_webapp restart # service dataprotection_webapp statu Refresh the ACM web page and login, it shows "Appliance Startup progress." It takes some time to resync up with all the appliance components, and once done it returns to the ACM dashboard. (This is not a process of restarting Appliance) If the issue still cannot be resolved, raise a support ticket with Dell Technologies. Avamar password related KB references: Avamar - How to restart AVE in single user mode to reset the root passwordIDPA: ChangeAvamarPasswords failed to perform change-passwords command on Avamar Utility Node

Support Cases