Symptoms

The GSAN was recently rolled back to a validated checkpoint, and now the MCS will not start. When tailing the tmp dpnctl mcs start output log, an error stating the gsan rollbacktime does not match stored rollbacktime was observed. This can apply when NetWorker VMware Protection integration is configured with the Virtual Backup Appliance (VBA) or the Integrated Data Protection Appliance (IDPA) or any Avamar Server. root@avamar:~/#: dpnctl status Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid) dpnctl: INFO: gsan status: up dpnctl: INFO: MCS status: down. dpnctl: INFO: Backup scheduler status: down. dpnctl: INFO: axionfs status: down. dpnctl: INFO: Maintenance windows scheduler status: enabled. dpnctl: INFO: Unattended startup status: enabled. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] root@avamar:~/#: tail -f /tmp/dpnctl-mcs-start-output-15545 Database server is running... INFO: Starting messaging service. INFO: Started messaging service. === BEGIN === check.mcs (prestart) check.mcs passed === PASS === check.mcs PASSED OVERALL (prestart) Starting Administrator Server... schedule_version key has value: 7.1.0.1, expecting value: 7.1.1.1. The Administrator Server preferences file needs to be updated. ERROR: gsan rollbacktime: 1495040451 does not match stored rollbacktime: 1477305907

Cause

The timestamp on the MCS and GSAN are off since the GSAN was rolled back to a previous point in time.

Resolution

Force an MCS restore so the MCS and GSAN are synced. root@labvba01:~/#: dpnctl start mcs --force_mcs_restore Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid) dpnctl: INFO: Restoring MCS data... dpnctl: INFO: MCS data restored. dpnctl: INFO: Starting MCS... dpnctl: INFO: To monitor progress, run in another window: tail -f /tmp/dpnctl-mcs-start-output-16158 dpnctl: WARNING: 1 warning seen in output of "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/mcserver.sh --start" dpnctl: INFO: MCS started. root@labvba01:~/#: dpnctl status Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid) dpnctl: INFO: gsan status: up dpnctl: INFO: MCS status: up. dpnctl: INFO: Backup scheduler status: down. dpnctl: INFO: axionfs status: down. dpnctl: INFO: Maintenance windows scheduler status: enabled. dpnctl: INFO: Unattended startup status: enabled. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] MCS is restored, now start backup scheduler. root@avamar:~/#: dpnctl start sched Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid) dpnctl: INFO: Resuming backup scheduler... dpnctl: INFO: Backup scheduler resumed. Validate the GSAN, MCS, and backup scheduler are up. root@labvba01:~/#: dpnctl status Identity added: /home/dpn/.ssh/dpnid (/home/dpn/.ssh/dpnid) dpnctl: INFO: gsan status: up dpnctl: INFO: MCS status: up. dpnctl: INFO: Backup scheduler status: up. dpnctl: INFO: axionfs status: down. dpnctl: INFO: Maintenance windows scheduler status: enabled. dpnctl: INFO: Unattended startup status: enabled. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"]

Support Cases