Symptoms

The GSAN is the main process that handles the incoming backup sessions from clients. The Avamar software must be shut down cleanly before any activities (such as applying a firmware update, security rollup, or scheduled power outage) are performed. This is vital to preserve the integrity of the data stored on the Avamar grid. During this shutdown process, services such as GSAN, Management Console Server (MCS - (the service responsible for the frontend Administrator UI) and so on are stopped. If the GSAN fails to shut down cleanly or becomes unresponsive during the shutdown process, a rollback is required when attempting to restart it. If this issue happens during a hotfix or workflow, the following message can be observed in the workflow.log (/data01/avamar/repo/temp/<package-name>/tmp/workflow.log) 2018-03-20 10:35:22 (+0000) 4107124 ERROR: Shutting down gsan failed - operation failed 2018-03-20 10:35:22 (+0000) 4107124 INFO: post: Shutting down gsan failed - operation failed 2018-03-20 10:35:22 (+0000) 4107124 INFO: ---------- END ShutdownGsan::do ------------ Several events in the logs (/data01/cur/gsan.log) can verify the issue (On a multinode grid the log is on the storage nodes). 1. Log in to the Avamar Utility Node as admin. 2. Load the ssh keys per Avamar: How to Log in to an Avamar Server and Load Various Keys. 3. Commands: The following command shows "Interrupted system call" messages: mapall --noerror 'grep -i "Interrupted system call" /data01/cur/gsan.log |tail -3' 2018/03/20-17:00:14.60114 {0.0} [acpt1-27000-172.22.4.168] tcpsockimpl::genAccept ::accept returned -1 signal=0 error=code 4: Interrupted system call 2018/03/20-17:00:14.60114 {0.0} [acpt2-29000-127.0.0.1] tcpsockimpl::genAccept ::accept returned -1 signal=0 error=code 4: Interrupted system call 2018/03/20-17:00:14.60114 {0.0} [acpt0-27000-127.0.0.1] tcpsockimpl::genAccept ::accept returned -1 signal=0 error=code 4: Interrupted system call 2018/03/20-17:00:14.60115 {0.0} [acpt3-29000-172.22.4.168] tcpsockimpl::genAccept ::accept returned -1 signal=0 error=code 4: Interrupted system call The following messages "shutdown already killed" are seen in a loop in the /data01/cur/gsan.log: mapall --noerror 'grep -i "shutdown.*<IP-address>" |tail -3' (Where "<IP-Address>" is the Bond0 IP Address of the Avamar Utility Node) Example where the Utility Node Bond0 IP Address is 10.10.10.168: mapall --noerror 'grep -i "shutdown.*10.10.10.168" |tail -3 ' 2018/03/20-20:11:52.41870 {0.0} [connbeat:216] servmain::shutdown already killed 10.10.10.168:56272 2018/03/20-20:12:02.41863 {0.0} [connbeat:216] servmain::checkconntimeout calling shutdown lastreqtime=22204826441151580 maxconninactive=3600 0x7fb36c5fb1e0 clientaddr=10.10.10.168:56272 ismaint=1 type=avmaint access=uname=root uid=0 priv=enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,superuser,ignoreacls,readdir,mclogin,opt1,opt2 avail=modes=00pu 2018/03/20-20:12:02.41867 {0.0} [connbeat:216] servmain::shutdown already killed 10.10.10.168:56272 ... 2018/03/20-20:12:12.41863 {0.0} [connbeat:216] servmain::checkconntimeout calling shutdown lastreqtime=22204826441151580 maxconninactive=3600 0x7fb36c5fb1e0 clientaddr=10.10.10.168:56272 ismaint=1 type=avmaint access=uname=root uid=0 priv=enabled,create,read,backup,access,move,delete,maint,manage,fullmanage,noticketrequired,superuser,ignoreacls,readdir,mclogin,opt1,opt2 avail=modes=00pu 2018/03/20-20:12:12.41866 {0.0} [connbeat:216] servmain::shutdown already killed 10.10.10.168:56272 The following command shows that there are still active tokens: mapall --noerror 'grep -i "tokendispenser::deactivate.*state=active" /data01/cur/gsan.log' 2018/03/20-17:00:14.60061 {0.0} [manage:628] tokendispenser::deactivate [state=active #tokens=3] The above message shows that there are still 3 active tokens that must be deactivated. 

Cause

This is a possible software bug (298711).

Resolution

Create a Service Request referencing this knowledge article.

Support Cases