Symptoms

IDPA Deployment may fail at 36% or 45% with error "FAILED: Configuring Protection Storage. Error: Unable to ping gateway server x.x.x.x using the IP address x.x.x.x configured on the ethVx network interface of the Protection Storage server."The deployment fails at 36% if the DD Interface ethV1 is unable to ping gateway. Deployment fails at 45% if the DD interface ethV2 is unable to ping the gateway.This issue may be present due to the switch configurations. Verify that Physical VMNIC ports 6 and 1 are connected to the switch and have single VLAN traffic that is allowed and configured in access mode. The following error may be seen on the IDPA ACM UI: In the following snippet, the deployment failed for ethV1. If the deployment fails on ethV2, the following error may be seen: Analyze the server.log on ACM to confirm that the following errors are present: 2020-02-26 14:17:39,934 INFO [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: checkConnectionWithGateway-->Non-exit code(41) while running ping.Output:[Ljava.lang.String;@56ce29822020-02-26 14:17:39,934 ERROR [pool-23-thread-1]-ddadapter.ConfigDataDomainTask: ApplianceException occurred while executing Datadomain config task.com.emc.vcedpa.common.exception.ApplianceException: Unable to ping the gateway server using the IP address configured on the ethV2 network interface of the Protection Storage server. Check the vSwitch1 connectivity on the IDPA Virtualization Server (ESXi server) and verify the switch settings in the customer environment. at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.checkConnectionWithGateway(ConfigDataDomainTask.java:242) at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.configureDD(ConfigDataDomainTask.java:330) at com.emc.vcedpa.ddadapter.ConfigDataDomainTask.run(ConfigDataDomainTask.java:1680) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748) Review the "/var/log/vobd.log" file on ESXi for the following error messages: [root@xxxxxxx:/var/log] grep -i 128 vobd.log2019-10-12T04:24:30.888Z: [netCorrelator] 133533756us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 1282019-10-12T04:24:30.889Z: [netCorrelator] 133534817us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 1282019-10-12T04:24:32.827Z: [netCorrelator] 135473005us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 1282019-10-12T04:28:03.941Z: [netCorrelator] 346586293us: [vob.net.pg.uplink.transition.down] Uplink: vusb0 is down. Affected portgroup: iDRAC Network. 0 uplinks up. Failed criteria: 1282019-10-12T04:34:10.179Z: [netCorrelator] 712824422us: [vob.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.2019-10-12T04:34:10.179Z: [netCorrelator] 712824515us: [esx.audit.net.firewall.config.changed] Firewall configuration has changed. Operation 'addIP4' for rule set snmpTrapsOut succeeded.2020-06-16T14:47:11.896Z: [netCorrelator] 25270540046us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: DP-appliance-mgmt. 1 uplinks up. Failed criteria: 1282020-06-16T14:47:11.896Z: [netCorrelator] 25270540052us: [vob.net.pg.uplink.transition.down] Uplink: vmnic7 is down. Affected portgroup: Management Network. 1 uplinks up. Failed criteria: 1282020-06-16T14:47:12.725Z: [netCorrelator] 25271369162us: [vob.net.pg.uplink.transition.down] Uplink: vmnic6 is down. Affected portgroup: DP-appliance-backup. 1 uplinks up. Failed criteria: 128 Make a note of the vmnic ID from the vobd.log showing above and run the following command to check firmware and driver version on the NIC: esxcli network nic get -n vmnicFor Example:[root@xxxxx4400:/var/log] esxcli network nic get -n vmnic7 Advertised Auto Negotiation: false Advertised Link Modes: 10000BaseT/Full Auto Negotiation: false Cable Type: DA Current Message Level: 0 Driver Info: Bus Info: 0000:3b:00:3 Driver: i40en Firmware Version: 6.80 0x80003d73 18.8.9 Version: 1.7.11 Link Detected: true Link Status: Up Name: vmnic7 PHYAddress: 0 Pause Autonegotiate: false Pause RX: false Pause TX: false Supported Ports: DA Supports Auto Negotiation: false Supports Pause: true Supports Wakeon: false Transceiver: Virtual Address: 00:50:56:53:d4:cf Wakeon: None

Cause

This issue may be present due to the switch configurations. Verify that Physical VMNIC ports 6 and 1 are connected to the switch and have single VLAN traffic that is allowed and configured in access mode. The network team should verify the switch configuration to verify that all four IDPA ports are set in access mode and have single VLAN allowed.Review the cabling and configuration guidelines below:For single network use case, review the information below: Ports 2,9,3,8 are connected to Management network VLAN ONLY (Single VLAN allowed).All ports on switch should be in "Access mode or untagged."All switch ports should be active and not configured in LACP.MTU size of 1518 or higher is configured on the external customer switch.MTU size of 1500 is configured along the network path. (Jumbo frames are not supported)If Cisco switches, port security is disabled over these ports. For separate management network use case, review the information below: Ports 2 and 9 are connected to Management network VLAN ONLY (Single VLAN allowed).Ports 3 and 8 are connected to Backup network VLAN ONLY (Single VLAN allowed).All ports on switch should be in "Access mode or untagged."All switch ports should be active and not configured in LACP.MTU size of 1518 or higher is configured on the external customer switch.MTU size of 1500 is configured along the network path. (Jumbo frames are not supported)If Cisco switches, port security is disabled over these ports.

Resolution

Note: This issue may be due to incorrect cabling and switch configuration issues at the switch end. Review cabling and configuration guidelines in 'Cause' section before proceeding. Resolution: 1. This issue may be due to incorrect cabling and switch configuration. Review KB article 192911: PowerProtect DP Series appliance and IDPA: DP4400 Networking and Initial Connectivity Resolution Path. to troubleshoot and fix DP4400 networking issues. 2. This issue may also be encountered if we use a duplicate IP address for backup interfaces ethV1 or ethV2 on Protection Storage (Data Domain). Perform ping tests to confirm if the IP is duplicate. Also verify there no stale records of this IP on Firewall, etc if this IP was in use earlier. 3. If the NIC i40en driver version is lower than 1.9.5, contact Dell EMC Support to update firmware. Also, if the i40en firmware version is lower than 19.5.12, contact Dell EMC Support to update firmware.

Support Cases