Symptoms

This KB helps to fix the issue with AVE Start Up in the event of a Power Outage or Unclean Shutdown of IDPA. This KB only applies to DP4400 Appliances. Please do not follow this KB article for other models. The Appliance Startup log shows the following Errors:(/data01/logs/Startup.log) [INFO] Setting DRS cluster behavior to manual [INFO] DRS cluster behavior set to manual [INFO] Starting DDVE [INFO] DDVE started successfully [INFO] Starting avamar [ERROR] Avamar start failed [INFO] Starting the IDPA vApp [INFO] IDPA vApp started successfully [INFO] Starting DPA Services [INFO] Starting DPS Services [INFO] Setting DRS cluster behavior to fullyAutomated [INFO] DRS cluster behavior set to fullyAutomated [INFO] Starting dashboard thread [INFO] Starting health monitor thread [INFO] Starting vSphere operation handler task If the appliance is still in the process of Starting up, the start up workflow may get hung at "Starting avamar" for ~ 2 hours. Eventually the Avamar Server Start up is marked as failed.Manually reviewing the services on the Avamar server, you may find that EMT and Backup Scheduler service may be down.On the Avamar Server, issue the following command: dpnctl status admin@xxxxxx:~/>: dpnctl statusIdentity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key)dpnctl: INFO: gsan status: updpnctl: INFO: MCS status: up.dpnctl: INFO: emt status: down.dpnctl: INFO: Backup scheduler status: down.dpnctl: INFO: Maintenance windows scheduler status: enabled.dpnctl: INFO: Unattended startup status: disabled.dpnctl: INFO: avinstaller status: up.dpnctl: INFO: ConnectEMC status: up.dpnctl: INFO: ddrmaint-service status: up.Additional Investigations are required to confirm we are hitting this Bug with Appliance Start Up.Verify the server.log on the ACM to confirm the following errors: Do you wish to do a restore of the local EMS data?^M ^M Answering y(es) will restore the local EMS data^M n(o) will leave the existing EMS data alone^M q(uit) exits with no further action.^M ^M Please consult with Avamar Technical Support before answering y(es).^M ^M Answer n(o) here unless you have a special need to restore^M the EMS data, e.g., you are restoring this node from scratch,^M or you know for a fact that you are having EMS database problems^M that require restoring the database.^M ^M ^M y(es), n(o), q(uit/exit): ] 2019-11-18 14:19:17,416 INFO [ApplianceStartupThread]-util.SSHUtil: STDERR : [] 2019-11-18 14:19:17,416 ERROR [ApplianceStartupThread]-util.SSHUtil: Failed to executed remote command using SSH 2019-11-18 14:19:39,055 INFO [ApplianceStartupThread]-util.SSHUtil: Remote command using SSH execution status: Host : [x.x.x.x] User : [admin] Password : [****************] Command : [dpnctl status all] STATUS : [1] 2019-11-18 14:19:39,055 INFO [ApplianceStartupThread]-util.SSHUtil: STDOUT : [] 2019-11-18 14:19:39,055 INFO [ApplianceStartupThread]-util.SSHUtil: STDERR : [Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key) dpnctl: INFO: gsan status: up dpnctl: INFO: MCS status: up. dpnctl: INFO: emt status: down. dpnctl: INFO: Backup scheduler status: down. dpnctl: INFO: Maintenance windows scheduler status: enabled. dpnctl: INFO: Unattended startup status: disabled. dpnctl: INFO: avinstaller status: up. dpnctl: INFO: ConnectEMC status: up. dpnctl: INFO: ddrmaint-service status: up. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]] This confirms the Appliance Start Up has issues where ACM is not handling the Avamar Prompt "Do you wish to do a restore of the local EMS data?" which causes the Appliance Start up to get hung for 2 hours and Avamar Start Up to be marked as failed.

Cause

This issue is permanently fixed in IDPA version 2.5.There is a patch/hotfix binary available for IDPA DP4400 Appliance with version 2.4 and 2.4.1. Please do not follow this KB for non DP4400 appliances. This fix is only applicable to IDPA version 2.4.

Resolution

Please contact IDPA Support and refer this article.

Support Cases