Symptoms

Error messages received from High Priority Events, ConnectEMC (Dial Home), or Events Management in the Management Console Server (MCS) UI: 22409 ERROR A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue. summary="A checkpoint validation (hfscheck) of Avamar Server checkpoint data is overdue." type="ERROR" code="22409" Symptom: 22409, Desc: A checkpoint validation (hfscheck) of server checkpoint data is overdue Checkpoints and Checkpoint validation (HFSCheck) do not run intermittently. This can be verified by running the following from the Avamar Utility Node: dumpmaintlogs --types=hfscheck,cp,gc --days=30 | grep '<4' | sort Sample output: 2014/09/30-15:01:02.71102 {0.0} <4200> starting scheduled garbage collection 2014/09/30-18:58:36.88121 {0.0} <4201> completed garbage collection 2014/10/01-15:01:00.64655 {0.0} <4200> starting scheduled garbage collection <...> 2014/10/12-15:00:58.45799 {0.0} <4200> starting scheduled garbage collection 2014/10/12-15:02:27.62869 {0.0} <4201> completed garbage collection 2014/10/13-15:01:04.91870 {0.0} <4200> starting scheduled garbage collection <...> 2014/10/16-15:01:00.49086 {0.0} <4200> starting scheduled garbage collection 2014/10/16-15:02:28.64102 {0.0} <4201> completed garbage collection 2014/10/17-15:01:00.71545 {0.0} <4200> starting scheduled garbage collection <...> The Waittime parameter is not set to the default time of 1 hour: This can be verified by running the following from the Avamar Utility Node, as admin with admin keys loaded. (See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys). mapall --noerror --parallel 'grep -h "crunchwaittask::body waittime=" /data01/cur/gsan.log*' | sort Sample output: 2014/10/12-15:02:27.63377 {0.0} [sched.crunchwait:4512] crunchwaittask::body waittime=11:57:33.000000 2014/10/13-15:02:23.17292 {0.0} [sched.crunchwait:3093] crunchwaittask::body waittime=00:00:00.000000 2014/10/14-15:02:27.37016 {0.0} [sched.crunchwait:5625] crunchwaittask::body waittime=00:00:00.000000 The Waittime parameter is not set properly on crunch.wait: This can be verified by running the following from the Avamar Utility Node: avmaint sched crunchwait --ava <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <task-param-request reset="false" task-type="crunchwait"> <crunchwait permanent="true"/>

Cause

The "crunchwait" parameter "waittime" is not set to the default of 60 minutes. 

Resolution

1. Log in to the Avamar Utility Node as admin and load the admin keys. See Avamar: How to Log in to an Avamar Server and Load Various Keys for instructions on loading keys. 2. Set the "waittime" to 60: avmaint sched crunchwait --waittime=60 --ava --permanent 3. Verify that the waittime is now present: avmaint sched crunchwait --ava <?xml version="1.0" encoding="UTF-8" standalone="yes"?> <task-params> <crunchwait permanent="true"> <param name="waittime" value="60"/> </crunchwait> </task-params> 4. Within the next day or two, after maintenance has finished, run the following command again: mapall --noerror --parallel 'grep -h "crunchwaittask::body waittime=" /data01/cur/gsan.log*' | sort The sample output should be similar to the following: 2014/10/14-15:02:27.37016 {0.0} [sched.crunchwait:5625] crunchwaittask::body waittime=00:00:00.000000 2014/10/15-15:02:24.63377 {0.0} [sched.crunchwait:4512] crunchwaittask::body waittime=01:00:00.000000 2014/10/16-15:03:01.17292 {0.0} [sched.crunchwait:3093] crunchwaittask::body waittime=01:00:00.000000 5. Monitor the grid over the coming days and verify that checkpoints and hfscheck are no longer being skipped.

Support Cases