Symptoms

Engineering introduced a new parameter called "page_fail_times_to_timeout_wo" in /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml file.When generating a WO (WorkOrder) for a client, MCS will page the client if Avamar server has the capacity. If the "page failure" number of times to the client is bigger than PAGE_FAIL_TIMES_TO_TIMEOUT_WO, then the client is considered offline.For this reason, MCS will timeout the WO directly.The recommend value of page_fail_times_to_timeout_wo is 10. If the page_fail_times_to_timeout_wo value is very big for example 9999, the job times out when it reaches its end of the schedule. Windows clients schedule backups remain in "Waiting-Client" and finish after around 6 minutes with "Timed Out - Start".On-Demand group backup works fine.In this specific scenario, the Avamar environment has only Linux and Windows clients backups.Affected version: 7.4.1-58 with MCS hotfix 302283 - hotfix-mc-cumulative-741-58-201809 or later7.5.0-183 with MCS hotfix 301047 - hotfix-mc-cumulative-v750-183-201810 or later7.5.1-101 with MCS hotfix 302282 - hotfix-mc-cumulative-751-101-201810 or later18.1.0-33 with MCS hotfix 304540 - hotfix-mc-cumulative-1810-33-201811 or later18.2.0-134 or later apply latest Cumulative update.

Cause

Since the implementation of the mentioned HFs on the affected versions listed above, MCS behaves differently for pageable clients: MCS generates a backup workorder for a client.If server has enough resource to start the job, it goes to "wait-client" status.If the client is pageable, MCS sends a wake-up message to the client.If the client does not respond to wake-up, MCS tries again (usually in 30 seconds)By default, MCS tried two times before it sets the job as timed out start NOTE: For more information about pageable clients, refer to the Avamar Administration Manual. NOTE: The change is made to recycle offline clients faster so that MCS would not allocate resource for those "offline" clients.

Resolution

There are two solutions for this issue: Disable automatic paging for the clients that are off the network or offline for extended periods of time. e.g., Desktop Laptop clients.Increase MCS wake up tries before setting the job as time-out start. Solution 1: How to Disable automatic paging for the clients that are off the network or offline for extended periods of time. Please refer to the Avamar Administration Manual - "Editing client paging settings" Solution 2:Apply latest MC cumulative update then if problem persists, perform following steps. How to Increase MCS wake up tries before setting the job as time-out start 1. Connect to Avamar server by SSH client (Putty) using. 2. Load the ssh keys of user:admin:ssh-agent bashssh-add ~admin/.ssh/admin_key 3. Make a backup for the mcserver.xml. cp -p /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/x-mcserver.xml.YYYYMMDD (where Y year, M month and D day) 4. Using the vi editor, edit the file /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml5. Search for 6. Add the following line in the end of this form: 7. Output should look like: admin@avamar:/usr/local/avamar/var/mc/server_data/prefs/>: grep page_fail_times_to_timeout_wo mcserver.xml 8. Save the file and restart MCS.dpnctl stop mcsdpnctl start mcsdpnctl start sched 9. Check if the next schedule backup works. Note: Contact Dell Avamar Technical Support if you are not comfortable to perform this change which requires a restart of MCS service.

Support Cases