Symptoms

During an upgrade, the workflow fails with one of the following errors: MCS failed to update the new catalog Updating Client Plug-in Catalog (nnn of nnn) Failed UpdateClientPluginCatalog|Updating Client Plug-in Catalog (nnn of nnn)|ERROR: UpdateClientPluginCatalog::do - operation failed The workflow.log (/data01/avamar/repo/temp/AvamarUpgrade*/tmp/workflow.log) contains more information about the error: tail -f workflow.log ... 2020-09-03 16:30:11 (+0530) 22794100 INFO: - - - - - - - END 2020-09-03 16:30:11 (+0530) 22794100 INFO: "rpm -qa --last | grep download-mcs-compat | cut --delimiter=' ' --fields=1 2>/dev/null", exit status=0 (success) 2020-09-03 16:30:11 (+0530) 22794100 INFO: download-mcs-compat-19.1.100-38-x86_64.rpm compare_rpms value: 0 2020-09-03 16:30:11 (+0530) 22794100 INFO: Rpm download-mcs-compat-19.1.100-38 is already installed on node 0.s 2020-09-03 16:30:11 (+0530) 22794100 INFO: post: Rpm download-mcs-compat-19.1.100-38 is already installed on node 0.s 2020-09-03 16:30:11 (+0530) 4107124 INFO: Processing rpm 1/1 succeeded 2020-09-03 16:30:11 (+0530) 4107124 INFO: post: Processing rpm 1/1 succeeded 2020-09-03 16:30:11 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=admin, ssh command is: (mcserver.sh --test --runasanyuser) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1 2020-09-03 16:30:15 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - - 2020-09-03 16:30:15 (+0530) 4107124 INFO: INFO: Administrator Server status: up 2020-09-03 16:30:15 (+0530) 4107124 INFO: - - - - - - - END 2020-09-03 16:30:15 (+0530) 4107124 INFO: "mcserver.sh --test --runasanyuser", exit status=0 (success) 2020-09-03 16:30:15 (+0530) 4107124 INFO: Informing the MC to use the new catalog 2020-09-03 16:30:15 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=root, ssh command is: (mccli plugin update --force) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1 2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - - 2020-09-03 16:30:18 (+0530) 4107124 INFO: 1,23996,CLI failed to connect to MCS. 2020-09-03 16:30:18 (+0530) 4107124 INFO: Attribute Value 2020-09-03 16:30:18 (+0530) 4107124 INFO: --------- ---------------------------------------------------- 2020-09-03 16:30:18 (+0530) 4107124 INFO: option AVAMAR001.corp.server.com: Name or service not known 2020-09-03 16:30:18 (+0530) 4107124 INFO: 2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - END 2020-09-03 16:30:18 (+0530) 4107124 ERROR: "mccli plugin update --force", exit status=1 (error) 2020-09-03 16:30:18 (+0530) 4107124 INFO: post: "mccli plugin update --force", exit status=1 (error) 2020-09-03 16:30:18 (+0530) 4107124 ERROR: MCS failed to update the new catalog 2020-09-03 16:30:18 (+0530) 4107124 INFO: post: MCS failed to update the new catalog 2020-09-03 16:30:18 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=admin, ssh command is: (/usr/local/avamar/bin/avhostname -f) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>/usr/local/avamar/var/run_command-syserr.14142.4107124 2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - BEGIN(output) - - - - - - - 2020-09-03 16:30:18 (+0530) 4107124 INFO: AVAMAR001.corp.company.com 2020-09-03 16:30:18 (+0530) 4107124 INFO: - - - - - - - END 2020-09-03 16:30:18 (+0530) 4107124 INFO: "/usr/local/avamar/bin/avhostname -f", exit status=0 (success) 2020-09-03 16:30:18 (+0530) 4107124 INFO: >>> [make_temp_script] 222 very_current_user=root, as_user=, ssh command is: (rpm -q dpnserver --last | head -1) >/usr/local/avamar/var/run_command-sysout.14142.4107124 2>&1 2020-09-03 16:30:18 (+0530) 4107124 INFO: Report home message: 2020-09-03 16:30:18 +0530|AVAMAR001.corp.server.com|19.1.0-38|AvamarUpgrade|19.1.0.38|/data01/avamar/repo/temp/AvamarUpgrade-19.1.0-38.avp_1599118779715/tmp|UpdateClientPluginCatalog|Updating Client Plug-in Catalog (172 of 226)|ERROR: UpdateClientPluginCatalog::do - operation failed 2020-09-03 16:30:18 (+0530) 4107124 ERROR: - operation failed 2020-09-03 16:30:19 (+0530) 4107124 INFO: post: - operation failed 2020-09-03 16:30:19 (+0530) 4107124 INFO: ---------- END UpdateClientPluginCatalog::do ------------ Both the hostname and mccli commands report the correct value: hostname -f AVAMAR001.corp.company.com mccli server show-prop 0,23000,CLI command completed successfully. Attribute Value -------------------------------------------- ---------------------------------- State Full Access Active sessions 0 Total capacity 3.9 TB Capacity used 101.2 GB Server utilization 2.6% Bytes protected (client pre-comp size) 62.9 GB Bytes protected quota (client pre-comp size) Not configured License expiration Never Time since Server initialization 365 days 02h:44m Last checkpoint 2020-09-03 13:36:24 IST Last validated checkpoint 2020-09-03 07:00:36 IST System Name servername System ID 1567585332@00:00:00:00:00:00 HFSAddr AVAMAR001.corp.company.com HFSPort 27000 IP address 10.x.x.87 Number of nodes 1 Nodes Online 1 Nodes Offline 0 Nodes Read-only 0 Nodes Timed-out 0

Cause

One of the following files has an incorrect mcsaddr value: /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml

Resolution

Note: For the purpose of this exercise, the correct hostname is AVAMAR001.corp.company.com. 1. As root, check which file has the wrong mcsaddr value: grep -i mcsaddr /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml <!-- mcsaddr : network name or IP of Administrator Server node --> mcsaddr="AVAMAR001.corp.company.com" grep -i mcsaddr /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml <!-- mcsaddr : network name or IP of Administrator Server node --> mcsaddr="AVAMAR001.corp.server.com" In this example, the incorrect version is /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml with a value of AVAMAR001.corp.server.com 2. Create a backup of the incorrect file: cp -p /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.`date +%y%m%d` -- Or -- cp -p /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml.`date +%y%m%d` 3. Edit the incorrect file: vi /root/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml -- Or -- vi /home/admin/.avamardata/var/mc/cli_data/prefs/mcclimcs.xml Update the incorrect mscaddr value. 4. Resume the workflow. If issues remain, create a swarm with the Avamar SER team.

Support Cases