Symptoms

User has changed or updated VPlex management server IP address(either cluster-1 or/both cluster-2) or cluster-witness IP address. Issue Description:Configuring a 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server fails with below error message: VPlexcli:/> configuration cw-vpn-configure -i xx.xx.xx.xx --force Please enter the IP address of the remote cluster management server that will be included in the 3-way VPN setup: yy.yy.yy.yy Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address yy.yy.yy.yy is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is not UP Cluster Witness Server at IP Address 128.221.254.3 is not reachable Error during Cluster Witness VPN Establishment: IPSEC configuration failed: Cannot proceed to configure the Cluster Witness Server IPSec configuration. It is possible that a 3-way VPN has already been established and you have given a wrong Cluster Witness Server public IP address. Please re-run the cluster witness server vpn configuration with the right public IP address. Resetting the Cluster Witness VPN configuration Resetting the Cluster Witness Server VPN configuration . . . <./truncated>

Cause

There could be two scenario's during which configuration of a 3-way VPN connection between VPlex management server (either cluster-1 or/both cluster-2) and cluster-witness server can fail as follows: User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server. AND/OR, User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server.

Resolution

Go through below scenario details and resolution step's in-order to resolve this issue: Scenario 1: User has changed VPlex management server IP address (eth3) (either cluster-1 or/both cluster-2) but cluster-witness server IPsec configuration file still contains an old IP address of the affected management server. NOTE: In the below example's, the user has changed the VPlex management server IP address of both cluster-1 and cluster-2. Gather correct IP address's assigned to VPlex management server (both cluster-1 and cluster-2) as follows: Cluster-1: VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14N.NNN.N.NNN is reachable Remote Internal Gateway addresses are reachable . </truncated> Cluster-2: VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14M.MMM.M.MMM is reachable Remote Internal Gateway addresses are reachable . </truncated> Perform SSH to the cluster-witness server using its public IP address: To find the public IP-address of the cluster-witness server, run the following VPlexcli command:Example: VPlexcli:/> ll /cluster-witness/ /cluster-witness: Attributes: Name Value ------------------ ------------- admin-state unknown private-ip-address 128.221.254.3 public-ip-address XX.XX.XX.XX <<< Cluster-Witness server public IP-address Perform SSH to the public IP-address of cluster-witness obtained from step 1.a as follows:service@ManagementServer:~> ssh <cluster-witness-public-IP-address>Example: service@ManagementServer:~> ssh xx.xx.xx.xx >> cluster-witness-public-IP-address Warning: Permanently added 'xx.xx.xx.xx' (ECDSA) to the list of known hosts. Last login: Mon Jun 06 15:33:14 2016 from xx.xx.xx.xx service@ClusterWitness:~> Cat the "IPsec.config" file and search for VPlex management server cluster-1 and cluster-2 IP address's as follows: Note: Before carrying out to step(3), confirm the actual IP address of the VPlex management server's from vpn status output using step(1). Once this information is gathered, compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not. Example: service@ClusterWitness:~> cat /etc/ipsec.conf # Add connections here. # Setup a tunnel between the management servers and the Cluster Witness Server # "left" means local, "right" means remote. # Connection between Cluster Witness Server and Management Server conn witness-cluster2 type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.254.3/32 leftcert=hostCert.pem right=15Y.YYY.Y.YYY <<========== Old/incorrect IP address of VPlex management server-2 rightsubnet=128.221.252.64/27,128.221.253.64/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start # Connection between Cluster Witness Server and Management Server conn witness-cluster1 type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.254.3/32 leftcert=hostCert.pem right=15X.XXX.X.XXX <<========== Old/incorrect IP address of VPlex management server-1 rightsubnet=128.221.252.32/27,128.221.253.32/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start As from the above example, we found that "IPsec.config" file of cluster-witness is still carrying the old IP-address of VPlex management server-1 and cluster-2. Therefore, edit the file "IPsec.config" of cluster-witness server using vi editor to update the correct IP-address of VPlex management server-1 and cluster-2. NOTE: Place the correct IP address of VPlex cluster-1 and cluster-2 after the equal sign with no space between the equal sign and IP address, save and exit the file. Example: service@ClusterWitness:~> vi /etc/ipsec.conf # Add connections here. # Setup a tunnel between the management servers and the Cluster Witness Server # "left" means local, "right" means remote. # Connection between Cluster Witness Server and Management Server conn witness-cluster1 type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.254.3/32 leftcert=hostCert.pem right=14M.MMM.M.MMM <<========== Add/update the correct IP address of VPlex cluster-1. rightsubnet=128.221.252.32/27,128.221.253.32/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKyyyyyyyyyyyy, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start # Connection between Cluster Witness Server and Management Server conn witness-cluster2 type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.254.3/32 leftcert=hostCert.pem right=14N.NNN.N.NNN <<========== Add/update the correct IP address of VPlex cluster-2. rightsubnet=128.221.252.64/27,128.221.253.64/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN: CKxxxxxxxxxxxx, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start Restart IPSEC service in both cluster-witness server and VPlex management servers (both cluster-1 and cluster-2) as follows: service@ClusterWitness:~> sudo /usr/sbin/ipsec restart service@ManagementServer:~> sudo /usr/sbin/ipsec restart Verify the status of IPsec service as follows: service@ClusterWitness:~> sudo /usr/sbin/ipsec status service@ManagementServer:~> sudo /usr/sbin/ipsec status Reperform the below command to reconfigure 3-way VPN connection between VPlex management server's and cluster-witness server as follows:Example:VPlexcli in Cluster-1: VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> force Then from VPlexcli in Cluster-2 VPlexcli:/> configuration cw-vpn-configure -i <cluster-witness-public-IP> force Verify the status of cluster-witness following step-1(both cluster-1 and cluster-2) as follows:Example: VPlexcli:/> ll /cluster-witness/* /cluster-witness/components: Name ID Admin State Operational State Mgmt Connectivity ----------------- -- ----------- ------------------- ----------------- cluster-1 1 enabled in-contact ok cluster-2 2 enabled in-contact ok server - enabled clusters-in-contact ok Check VPN connectivity using command vpn status(both cluster-1 and cluster-2) as follows: Example:Cluster-1: VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14N.NNN.N.NNN is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable Cluster-2: VPlexcli:/> vpn status Verifying the VPN status between the management servers... IPSEC is UP Remote Management Server at IP Address 14M.MMM.M.MMM is reachable Remote Internal Gateway addresses are reachable Verifying the VPN status between the management server and the cluster witness server... IPSEC is UP Cluster Witness Server at IP Address 128.221.254.3 is reachable Scenario 2: User has changed cluster-witness IP address but VPlex management server(either cluster-1 or/both cluster-2) IPsec configuration file still contains old IP address of cluster-witness server. Verify the correct cluster-witness public IP-address by following the below command: VPlexcli:/> ll /cluster-witness/** /cluster-witness: Attributes: Name Value ------------------ ------------- admin-state enabled private-ip-address 128.221.254.3 public-ip-address xx.xx.xx.65 <<< Cluster-Witness server public IP-address Contexts: Name Description ---------- -------------------------- components Cluster Witness Components From the VPlex management server(either cluster-1 or cluster-2) cat the file ipsec.conf as follows: Note: Before carrying out to step(10), confirm the actual IP address of the VPlex management server's from vpn status output using step(9). Once this information is gathered, compare it with the "IPsec.config" file mentioned below to check/confirm if it is matching or not. Example: service@Managementserver:~> cat /etc/ipsec.conf >> Cluster-1 # Add connections here. # Setup a tunnel between the management servers and their networks # "left" means local, "right" means remote. # Connection between Cluster Witness Server and Management Server conn net-witness type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.252.64/27,128.221.253.64/27 leftcert=hostCert.pem right=xx.xx.xx.45 <<========== Old/incorrect IP address of cluster-witness rightsubnet=128.221.254.3/32 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN CWS, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes128-sha1 auto=start # Connection between Management Server 1 and Management Server 2 conn net-net type=tunnel keyexchange=ikev2 mobike=no reauth=no left=%defaultroute leftsubnet=128.221.252.64/27,128.221.253.64/27 leftcert=hostCert.pem right=14N.NNN.N.NNN <<========== IP address of remote management server rightsubnet=128.221.252.32/27,128.221.253.32/27 rightid="C=US, ST=Massachusetts, O=EMC, OU=EMC, CN=VPlex VPN:CKxxxxxxxxxxxx, E=support@emc.com" ike=3des-sha256-modp2048 esp=aes256-sha256 auto=start As from the above example, we found that "IPsec.config" file of VPlex management server (either cluster-1 or/both cluster-2) is still carrying the old IP-address of cluster-witness server. Therefore, edit the file "IPsec.config" using vi editor to update the correct IP-address of the cluster-witness server. NOTE: Place the correct IP address of Cluster-witness server on the affected VPlex management server after the equal sign with no space between the equal sign and IP address, save and exit the file. Repeat step's from 4 to 8 from scenario 1 in-order to resolve this issue.

Support Cases