Symptoms

This has been observed when EMS or MCS restores have been performed, a node was replaced, or a rollback performed.After bringing down the Avamar services to perform maintenance, all the services could come back online apart from EMS and DTLT. dpnctl status Identity added: /home/admin/.ssh/dpnid (/home/admin/.ssh/dpnid) dpnctl: INFO: gsan status: up dpnctl: INFO: MCS status: up. dpnctl: INFO: EMS status: down. dpnctl: INFO: Backup scheduler status: down. dpnctl: INFO: dtlt status: down. dpnctl: INFO: Maintenance windows scheduler status: enabled. dpnctl: INFO: Unattended startup status: enabled. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] The DPN control log (/usr/local/avamar/var/log/dpnctl.log) shows the following sequence of events and reports an error: less /usr/local/avamar/var/log/dpnctl.log 2017/11/02-12:52:56 dpnctl: INFO: - - - - - - - - - - - - - - - BEGIN 2017/11/02-12:52:56 2017/11/02-12:52:56 no element found at line 1, column 0, byte 0: 2017/11/02-12:52:56 2017/11/02-12:52:56 ^ 2017/11/02-12:52:56 at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187 2017/11/02-12:52:56 dtltctl: INFO: Desktop/laptop services status: down. 2017/11/02-12:52:56 dpnctl: INFO: - - - - - - - - - - - - - - - END 2017/11/02-12:52:56 /bin/cat /tmp/dpnctl-dtlt-status-status-30477 2>&1 2017/11/02-12:52:56 [ "/bin/cat /tmp/dpnctl-dtlt-status-status-30477 2>&1" exit status = 0 ] 2017/11/02-12:52:56 dpnctl: INFO: "[ -r /etc/profile ] && . /etc/profile ; /usr/local/avamar/bin/../lib/dpnutils/dtltctl status" exit status = 1 2017/11/02-12:52:56 rm -f /tmp/dpnctl-dtlt-status-status-30477 /tmp/dpnctl-dtlt-status-output-30477 2017/11/02-12:52:56 dpnctl: INFO: "rm -f /tmp/dpnctl-dtlt-status-status-30477 /tmp/dpnctl-dtlt-status-output-30477" - exit status 0 2017/11/02-12:52:56 dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] 2017/11/02-12:52:56 rm -f /usr/local/avamar/var/dpnctl.lck 2017/11/02-12:52:56 ======= dpnctl 7.0.2-43 (1.139), running as admin, RETURNING at 2017-11-02 18:22:56 IST ======= 2017/11/02-12:52:56 [user "admin"] program (pid 30477) exit status = 1 (error) 2017/11/02-12:52:56 ======= dpnctl 7.0.2-43 (1.139), running as admin, ENDING at 2017-11-02 18:22:56 IST ======= 2017/11/02-12:52:56 [user "admin"] program (pid 30403) exit status = 1 (error) Attempting a manual flush of the EMS database fails with the error "no element found at line 1, column 0, byte 0:" emserver.sh --flush no element found at line 1, column 0, byte 0: ^ at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187 An attempt to start the EMS service with the "verbose" flag may reveal the cause. emserver.sh --start --verbose args = --start --verbose --------------------------------- home_dir=/usr/local/avamar db_dir=/usr/local/avamar/var/em/server_data/postgres/data data_dir=/usr/local/avamar/var/em/server_data postgres_dir=/usr/local/avamar/var/em/server_data/postgres prefs_dir=/usr/local/avamar/var/em/server_data/prefs mc_dir=/usr/local/avamar/var/mc log_dir=/usr/local/avamar/var/em/server_log lib_dir=/usr/local/avamar/lib bin_dir=/usr/local/avamar/bin --------------------------------- Processing /usr/local/avamar/lib/emserver.xml Processing /usr/local/avamar/var/em/server_data/prefs/emserver.xml no element found at line 1, column 0, byte 0: ^ at /usr/lib/perl5/vendor_perl/5.10.0/x86_64-linux-thread-multi/XML/Parser.pm line 187

Cause

The EMS service configuration file /usr/local/avamar/var/em/server_data/prefs/emserver.xml is empty. ls -thrl /usr/local/avamar/var/em/server_data/prefs/ | grep emserver -rwxrwx--- 1 admin admin 0 Aug 10 16:05 emserver.xml The EMS service configuration file /usr/local/avamar/var/em/server_data/prefs/emserver.xml is malformed. admin@avamar:/usr/local/avamar/var/em/server_data/prefs> ls -l emserver.xml -rw-r----- 1 admin admin 9497 Sep 27 2021 emserver.xml admin@avamar:/usr/local/avamar/var/em/server_data/prefs> cat emserver.xml admin@avamar:/usr/local/avamar/var/em/server_data/prefs> admin@avamar:/usr/local/avamar/var/em/server_data/prefs> vi emserver.xml ^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@^@

Resolution

Must find a valid copy of emserver.xml from an old EM_BACKUPS flush.Use avtar to find EM_BACKUPS flushes from the past.A count of flushes to list can be specified in case there are thousands.Leave out the count flag to list all EM_BACKUPS. avtar --backups --path=/EM_BACKUPS --count=500 Select a label number and restore to a temporary directory to inspect the old emserver.xml copy. mkdir ~admin/em_redirected_restore cd ~admin/em_redirected_restore avtar -x --path=/EM_BACKUPS --labelnum= --target=. Inspect the emserver.xml copy to see if it is valid. cat var/em/server_data/prefs/emserver.xml Optionally use xmllint to validate the xml file. xmllint var/em/server_data/prefs/emserver.xml --noout Once the emserver.xml file from an old flush has been validated, copy it to the proper location. cp ~admin/em_redirected_restore/var/em/server_data/prefs/emserver.xml /usr/local/avamar/var/em/server_data/prefs/emserver.xml Update permissions and ownership. chown admin:admin /usr/local/avamar/var/em/server_data/prefs/emserver.xml chmod 640 /usr/local/avamar/var/em/server_data/prefs/emserver.xml Start EMT. dpnctl start emt

Support Cases