Symptoms

Overview During MX9116n reboot, and when server traffic falls back to the newly reloaded I/O Module (IOM), transient traffic loss may be observed.Affected Platforms MX7116n Fabric Expander Module (FEM)Affected ConfigurationsMX9116n running in Full Switch modeAffected Firmware10.5.0.X and later versionsNomenclature used in this articleIOM: Ethernet I/O Module MX9116nFEM: Fabric Expander Module MX7116nVirtual ports: Internal ports of the FEM when is mapped to an IOMPhysical ports: IOM external ports connected to a FEMUFD: Uplink Failure Detection

Cause

During IOM boot up, there is a delay between the virtual FEM port coming up and the mapping to the physical port on the IOM.For a short period of time (40 seconds), any traffic received on the internal FEM ports may not be forwarded out through the IOM physical ports.This packet loss behavior may be observed in IOMs operating in Full Switch Mode and server's NICs in switch-independent bonding with Failback enabled.

Resolution

Follow one of the following solutions: 1. IOM with FEMs On-Boarded, configured with UFD (Preferred Method)2. IOM with FEMs On-Boarded, configured with VLT Delay-restore Orphan-port3. IOM with FEMs not On-Boarded (Pass-Through mode)4. IOM operating in SmartFabric mode5. Hosts with higher NIC Teaming failback delay 1. IOM with FEMs On-boarded, configured with UFD (Preferred Method) In this example, the FEMs are onboarded on the connected IOM. Figure 1: Show unit provisioning on-boarded. Figure 2: Show running | grep unit. The port group numbers vary based on the cabling between the MX9116n and the FEM. In our case, the Port Groups in use are: port-group 1/1/1, port-group 1/1/3 Figure 3: Show discovered-expanders The Physical ports used are: Figure 4: Show port-group port-group 1/1/1 => Ports 1/1/17:1-1/1/17:4 & 1/1/18:1-1/1/18:4 port-group 1/1/3 => Ports 1/1/21:1-1/1/21:4 & 1/1/22:1-1/1/22:4 The relative Virtual ports used are: Unit ID 71 => Ports 1/71/1-1/71/8 Unit ID 72 => Ports 1/72/1-1/72/8 Mapping example for connected Host on port 1/71/3 Figure 5: Show interface for Physical Port Create a UFD group and configure both Physical and Virtual ports as downstream.UFD Configuration example for connected Host on port 1/71/3 Figure 6: Show running UFD. A similar configuration must be applied on the other IOM. 2. IOM with FEMs On-Boarded, configured with VLT Delay-restore Orphan-port Apply the following command on the Physical and the Virtual ports. VLT Delay-restore Orphan-port enable Configuration example for connected Host on port 1/71/3 Figure 7: Show running Virtual Int. Figure 8: Show running Physical Int. Verify the configuration with the command: show vlt X delay-restore-orphan-ports Figure 9: Show vlt X delay-restore-orphan-ports. NOTE: To identify the ports to be configured, follow Solution 1. A similar configuration must be applied on the other IOM. 3. IOM with FEMs not On-Boarded (Pass-Through mode) Do not configure Virtual ports on FEMs. This is accomplished by NOT using the command unit provision. Figure 10: Show unit provisioning NOT on-boarded. Apply the switch config directly on the physical ports.Configuration example for connected Host on port 1/71/3 Figure 11: Show running Physical Int. NOTE: UFD and "VLT Delay-restore Orphan-port enable" are not required in this configuration, but UFD is still recommended for Uplink Failure Protection. A similar configuration must be applied on the other IOM. 4. IOM operating in SmartFabric mode IOMs, operating in SmartFabric mode, are not affected. 5. Hosts with higher NIC Teaming failback delay Increase the failback delay time to a minimum of 30 seconds.Example for ESXi Hosts:Change the Net.TeamPolicyUpDelay value to 30 seconds or any higher value as recommended.The procedure to modify the Up-Delay parameter is detailed in the VMware article Changing the network teaming failback delay fails in ESXi (2014075) . NOTE: For equivalent configuration in other Operating Systems, contact the respective vendors.

Support Cases