Symptoms

New storage nodes are added to Avamar. It is a replication target.A replication job which used to work starts to fail intermittently. The following messages are found in the replication logs 2014-11-19 22:10:42 avtar Info : Connecting to Avamar Server (avamar-source) 2014-11-19 22:10:42 avtar Info : Connecting to one node in each datacenter 2014-11-19 22:10:42 avtar Info : - Connect: Connected to 10.46.76.180:27000, Priv=0, TCP 2014-11-19 22:10:42 avtar Info : - Datacenter 0 has 1 nodes: Connected to 10.46.76.180:27000, Priv=0, TCP 2014-11-19 22:10:42 avtar Info : Connecting to Avamar Server (avamar-target) 2014-11-19 22:10:42 avtar Info : Connecting to one node in each datacenter 2014-11-19 22:10:47 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:10:52 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:10:57 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:02 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:07 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:12 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:17 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:22 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:27 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:32 avtar Info : - Failed initial handshake, trying again 2014-11-19 22:11:32 avtar Info : - Connect: Connected to 10.14.3.30:27000, Priv=0, TCP 2014-11-19 22:11:33 avtar Info : - Trying connection 10.14.3.35:27000 2014-11-19 22:11:39 avtar Info : - Trying connection 10.14.3.38:27000 2014-11-19 22:11:44 avtar Info : - Trying connection 10.14.3.44:27000 2014-11-19 22:11:49 avtar Info : - Trying connection 10.14.3.39:27000 2014-11-19 22:11:55 avtar Info : - Trying connection 10.14.3.36:27000 2014-11-19 22:12:00 avtar Info : - Trying connection 10.14.3.43:27000 2014-11-19 22:12:05 avtar Info : - Trying connection 10.14.3.37:27000 2014-11-19 22:12:11 avtar Info : - Trying connection 10.14.3.42:27000 2014-11-19 22:12:16 avtar Info : - Trying connection 10.14.3.44:27000 2014-11-19 22:12:21 avtar Info : No connections available 2014-11-19 22:12:21 avtar FATAL : Fatal server connection problem, aborting initialization. Verify correct server address and login credentials and that your server supports un-encrypted network connections. 2014-11-19 22:12:21 avtar FATAL : Failed to initiate session with servers for replication. 2014-11-19 22:12:21 avtar Info : Error summary: 2 errors: 8604, 10776 2014-11-19 22:12:21 avtar Info : Sending wrapup message to parent 2014-11-19 22:12:21 avtar Info : Command failed (2 errors, exit code 10008: cannot establish connection with server (possible network or DNS failure)) As can be seen in the logs, the Avamar source has a single node.The replication job can connect to the Avamar target using IP 10.1.1.30.It cannot reach nine of the nodes presented to it by the dispatcher table. The target system IP address has been confirmed to be correctThe login credentials on the target are correct

Cause

When the Avamar target was expanded, the necessary firewall ports to enable connectivity to the new nodes were not opened.When the replication connects to any original storage nodes, it succeeds.When the replication connects to new storage nodes, it fails.

Resolution

Check network connectivity between all nodes on the Avamar source and all nodes on the target system. For more information, see the Avamar Product Security guide. https://www.dell.com/support/home/en-us/product-support/product/avamar-server/docs​​​​​​​

Support Cases