Symptoms

Whilst waiting for a secure progress message to connect to the Control (CTL) port, random scheduled VMware image jobs get stuck or stop responding. ------------------------------------------------------------------- ----- START avvcbimage [19.3.100-149_HF333148 Linux-x86_64] ------------------------------------------------------------------- avvcbimage Info : Logging to /usr/local/avamarclient/var/-1016-vmimagel.log avvcbimage Info : - Reading /usr/local/avamarclient/var/avvcbimage.cmd avvcbimage Info : - Reading /usr/local/avamarclient/var/avvcbimageAll.cmd avvcbimage Info : CTL listening on port 36461 Nothing else happens until the job is cancelled.

Cause

The Avamar Server GSAN is rejecting sessions ticket login due to expired session ticket. Background: For VM, SQL and other plugins, Avamar's avagent service spawns two process at the start of a backup or restore job. The plug-in process (Example avvbimage or avsql) The avtar secure progress workorder process is used to obtain the workorder for the plug-in. The avagent.log shows the following pattern: avagent Info : Requesting work from avagent Info : Process 6373 (/usr/local/avamarclient/bin/avvcbimage) for workorder MOD-1655480090808 started avagent Info : Using secure session tickets and progress avtar for workorder processing avagent Info : Process 6376 (/usr/local/avamarclient/bin/avtar.bin) for workorder progress-MOD-1655480090808 started Hidden log: The Avamar UI does not fetch the avtar secure progress log. For VM backup logs, this file is located in /usr/local/avamarclient/var/clientlogs.An example of a log file name is; /usr/local/avamarclient/var/clientlogs/progress---1001-Unix.log This avtar progress log shows a failed login to Avamar. This login failure prevents a connection back to the plug-in Control (CTL) ports. The plug-in may stop responding. 02:28:36 avtar Info : Initializing connection 02:28:36 avtar Info : Connecting to Avamar Server (avamareserver.example.com) 02:28:36 avtar Info : Connecting to one node in each datacenter 02:31:17 avtar Info : - Connect: Connected to 1.2.3.4:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384 02:35:20 avtar Info : - Datacenter 0 has 1 nodes: Connected to 1.2.3.4:29000, Priv=0, SSL Cipher=ECDHE-RSA-AES256-GCM-SHA384 02:35:20 avtar Info : Logging in on connection 0 with Session Ticket 02:35:20 avtar Info : Using Secure Session Ticket Format 02:35:20 avtar Info : Starting graceful (staged) termination, Fatal server error (wrap-up stage) 02:35:20 avtar Error : Connection killed from GSAN. 02:35:20 avtar FATAL : Fatal Server Error occurred (MSG_ERR_AUTH_FAIL), aborting execution (SECURETICKETLOGIN=452 serial=1 seq=0 flags=R:H:0 kind=0 rsp=MSG_ERR_AUTH_FAIL) 02:35:20 avtar Error : Login error 5: Authorization failure (Session Ticket login) 02:35:20 avtar FATAL : Fatal server connection problem, aborting initialization. Verify correct server address and login credentials. 02:35:20 avtar Info : Error summary: 4 errors: 8941, 5704, 5126, 8609 02:35:20 avtar Info : Progress avtar, no wrapup. 02:35:20 avtar Info : Command failed (4 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure)) Here, the session ticket expiration (session_ticket_expiration_minutes) was using the default value of 5 minutes. The avtar connection took 6 minutes and 44 s to establish a connection. admin@ave198:~/>: grep session_ticket_expiration_minutes /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml

Resolution

On the Avamar server, increase the session_ticket_expiration_minutes to 10 min (or more) by following Avamar: Backups to Avamar fail with - avtar Error : Login error 5: Authorization failure (Session Ticket login)

Support Cases