Symptoms

The avinstaller.log.0 shows the NVE upgrade completed step 13, with exit status 0, to enable the zzz-avi service. It then proceeds to step 14 of 45 which is rebooting the NVE: ... Month DD, YYYY HH:MM:SS com.avamar.avinstaller.process.InstallManager completeTask INFO: ExitCode: 0 Month DD, YYYY HH:MM:SS com.avamar.avinstaller.process.InstallManager completeTask INFO: Completed task: Enable zzz-avi service for startup (13 of 45) id: 20153 Month DD, YYYY HH:MM:SS com.avamar.avinstaller.process.InstallManager completeTask INFO: Working on task: Rebooting Nve Node (14 of 45) id: 20157 Month DD, YYYY HH:MM:SS com.avamar.avinstaller.monitor.PollMessageSender sendMessage INFO: PollMessageSender sent: 27Rebooting Nve Node (14 of 45)nve-upgradeYYYYMMDD081628320943nve-upgradeYYYYMMDD1628320943.2000720157YYYY/MM/DD-HH:MM:SS.00175 ... The "avinstaller.pl --start" or "systemctl start avi.service" shows: root@my-nve-name://#: avinstaller.pl --start Avistart process: INFO: Starting AVI service... Job for avi.service failed because the control process exited with error code. See "systemctl status avi.service" and "journalctl -xe" for details. ERROR: Failed to start AVI service. root@denqcb3nwve01://#: systemctl status avi.service ● avi.service - Avamar Installation Manager Loaded: loaded (/usr/lib/systemd/system/avi.service; enabled; vendor preset: disabled) Active: failed (Result: exit-code) since YYYY-MM-DD HH:MM:SS MDT; 1min 2s ago Process: 30542 ExecStart=/usr/local/avamar/bin/avinstaller.pl --daemon (code=exited, status=1/FAILURE) Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Local backup dir: /usr/local/avamar/src/lockbox_backup/YYYY-MM-DD-HH_MM Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Update avi password to avlockbox successfully. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Check if AvInstaller is required to be upgraded... Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Checking keystore type. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: ERROR: Error happened when getting keystore passphrase from avlockbox Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Control process exited, code=exited status=1 Month DD HH:MM:SS my-nve-name systemd[1]: Failed to start Avamar Installation Manager. Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Unit entered failed state. Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Failed with result 'exit-code'. The "journalctl -xe" output shows: root@my-nve-name://#: journalctl -xe Month DD HH:MM:SS my-nve-name cron[30254]: pam_unix(crond:session): session opened for user root by (uid=0) Month DD HH:MM:SS my-nve-name CRON[30254]: pam_unix(crond:session): session closed for user root Month DD HH:MM:SS my-nve-name su[30432]: (to root) admin on pts/0 Month DD HH:MM:SS my-nve-name su[30432]: pam_unix(su-l:session): session opened for user root by admin(uid=500) Month DD HH:MM:SS my-nve-name liblogging-stdlog[901]: -- MARK -- Month DD HH:MM:SS my-nve-name cron[30496]: pam_unix(crond:session): session opened for user root by (uid=0) Month DD HH:MM:SS my-nve-name CRON[30496]: pam_unix(crond:session): session closed for user root Month DD HH:MM:SS my-nve-name systemd[1]: Starting Avamar Installation Manager... -- Subject: Unit avi.service has begun start-up -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit avi.service has begun starting up. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Avistart process: Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: /usr/local/avamar/bin/avlockbox.sh Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Lockbox /usr/local/avamar/var/lockbox/avlockbox.clb file exists Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Avlockbox db exists. Month DD HH:MM:SS my-nve-name passwd[30553]: pam_unix(passwd:chauthtok): password changed for avi Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: New password: Retype new password: passwd: password updated successfully Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Set password of avi user successfully. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Try to update avi password. It's possible that avi password have already exist in avlockbox. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup lockbox file Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup keystore files Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup SSV files Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Flush backup Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Local backup dir: /usr/local/avamar/src/lockbox_backup/YYYY-MM-DD-HH_MM Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Updated with new value under name "avi". Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup lockbox file Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup keystore files Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Backup SSV files Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Flush backup Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Local backup dir: /usr/local/avamar/src/lockbox_backup/YYYY-MM-DD-HH_MM Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Flush backup dir: /usr/local/avamar/var/mc/server_data/lockbox_backup Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Update avi password to avlockbox successfully. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: INFO: Check if AvInstaller is required to be upgraded... Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: Checking keystore type. Month DD HH:MM:SS my-nve-name avinstaller.pl[30542]: ERROR: Error happened when getting keystore passphrase from avlockbox Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Control process exited, code=exited status=1 Month DD HH:MM:SS my-nve-name systemd[1]: Failed to start Avamar Installation Manager. -- Subject: Unit avi.service has failed -- Defined-By: systemd -- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel -- -- Unit avi.service has failed. -- -- The result is failed. Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Unit entered failed state. Month DD HH:MM:SS my-nve-name systemd[1]: avi.service: Failed with result 'exit-code'. The "ask_pass -s keystore_passphrase changeme" and "avlockbox.sh -r keystore_passphrase changeme" commands generate: vvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvvv An error has occured: The Lockbox stable value threshold was not met because the system fingerprint has changed. To reset the system fingerprint, open the Lockbox using the passphrase. ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

Cause

The NetWorker Installation Manager avlockbox uses the operating system (operating system) kernel fingerprint. The NveUpgrade-19.4.0-126.avp package performs an OS upgrade and there is a mismatch in the avlockbox and operating system fingerprint. The cause of the mismatch fingerprint is not known.

Resolution

The avlockbox should be re-created with the new operating system fingerprint. 1. Shut down the NVE and take a Virtual Machine snapshot in the vSphere Web Client.2. Power on the NVE.3. Open a Secure Shell (SSH) connection and login as admin.4. Sudo to the root user: sudo su - 5. Re-create the avlockbox with the "Init lockbox file" and "Force init lockbox file" options: /usr/local/avamar/bin/avlockbox.sh -i -f 6. Update the avlockbox keystore_passphrase: /usr/local/avamar/bin/avlockbox.sh -s keystore_passphrase changeme 7. Create a copy of the avinstaller keystore: mv /usr/local/avamar/lib/avi/avi_keystore /usr/local/avamar/lib/avi/x-avi_keystore.`date -I`.bak 8. Re-create the keystore with a new certificate: keytool -keystore /usr/local/avamar/lib/avi/avi_keystore -genkey -alias tomcat -keyalg RSA -sigalg SHA512withRSA -keysize 3072 -keypass changeme -storepass changeme -validity 3650 -dname "CN=`hostname`, OU=Dell EMC, O=Dell Technologies, L=Irvine, S=California, C=US" 9. Confirm that the new avi_keystore was created as PKCS12 type, and there is a tomcat certificate generated: keytool -list -keystore /usr/local/avamar/lib/avi/avi_keystore -storepass changeme Example: nve:~ # keytool -list -keystore /usr/local/avamar/lib/avi/avi_keystore -storepass changeme Keystore type: PKCS12 Keystore provider: JsafeJCE Your keystore contains 1 entry tomcat, Aug 17, 2023, PrivateKeyEntry, Certificate fingerprint (SHA-256): 0A:FD:41:0A:5A:D0:6D:D1:EE:50:61:63:81:F0:A2:9B:19:DD:B7:F2:6F:00:F9:1C:45:DB:81:0D:49:30:89:63 10. Start the avi.service: avinstaller.pl --start 11. Validate the avi.service status: systemctl status avi.service 12. Monitor the remainder of the upgrade using the NetWorker Installation Manager.13. Remove the Virtual Machine snapshot once the upgrade is complete.

Support Cases