Loading...
Loading...
VersionRelease DateDetails206/30/2021Added a note to the Resolution section.106/22/2021Original Document Release.An issue has been found in M-series environments that utilize Multi-Chassis Link Aggregation (MLAG) where switches in an MLAG pair remain in a peering state for an extended period of time after being rebooted or physically power-cycled.The source of the issue is a default MTU change to 9216 bytes included in ONYX-3.9.2110 M-Series switch firmware. This issue is exacerbated after upgrading from an earlier version than ONYX-3.9.2110 to ONYX-3.9.2110; however later versions of ONYX have corrected this issue. The original intent was to change the default L2/L3 value of all switch ports from the MTU default 1500 bytes to the maximum allowed value of 9216 bytes to allow for better performance in the event of data flow through the IPL. It has been found that a conflict affecting MLAG operation arises when there is a MTU 1500 force configuration statement visible in the running configuration file.The MTU of the port channel and ports of the inter-peer link (IPL) will not automatically be configured to 9216 bytes if they are configured with theforcecommand prior to the upgrade. This will lead to an inconsistency between the ports and the port channel and prevent the MLAG pair from synchronizing, which will furthermore remove any hardware redundancy in the event of a failure.
The following HPE Storage Networking M-Series ONYX switches are affected by this issue:HPE M-series SN2010M Switch Q9E63AHPE M-series SH2200M Switch for SynergyHPE M-series SN2100M Switch Q2F23A and Q2F24AHPE M-series SN2410M Switch Q2F22A and Q6M27AHPE M-series SN2410bM Switch Q6M28A and Q6M29AHPE M-series SN2700M Switch Q2F21A and Q6M26AHPE M-series SN3700M and SN3700cM Switch R5Z74A and R3B14AHPEM-series SN4600cM Switch R6R24A
The way to pinpoint an IPL MTU mismatch is to verify whether packets on the IPL ports are getting discarded for any reason. In the following example, ports 30 and 31 are part of port-channel 1 (Po1) which is assigned to the IPL:Run these CLI commands on both switches in the MLAG pair.SN3700C-ONYX-ODD1 [vip-4243: master] (config) # show interfaces port-channel summaryFlagsD: DownU: UpP: Up in port-channel (members)S: Suspend in port-channel (members)I: Individual------------------------------------------------------------------------Group Port-Channel Type Member Ports------------------------------------------------------------------------1 Po1(U) LACP Eth1/30(P) Eth1/31(P)2 Po3(U) LACP Eth1/11(P) Eth1/12(P)Eth1/13(P) Eth1/14(P)HPE now initiates a command to verify the number of discards seen and the reasons. A correctly configured MLAG IPL should not be experiencing large numbers of discards under any circumstance.SN3700C-ONYX-ODD1 [vip-4243: master](config) # show interfaces ethernet 1/30 counters discardInterface Eth1/30:Rx:50000 discard packets22110 error packets14211 fcs errors0 undersize packets12002 oversize packetsTx:47833 discard packets20211 error packets0 hoq discard packets14321 oversize packetsWhen HPE examines the system configuration with theshow running-configcommand and compare it to the output ofshow interfaces port-channel 1,then the source of the MTU mismatch can be found:## Interface Ethernet configuration##interface ethernet 1/30-1/31 mtu 1500 forceinterface ethernet 1/30-1/31channel-group 1 mode activeSN3700C-ONYX-ODD1 [vip-4243: master] (config) # show interfaces port-channel 1Po1:Admin state : EnabledOperational state : UpDescription : IPL to other switchMac address : N/AMTU : 9216 bytes (Maximum packet size 9238 bytes)The ports comprising Port Channel 1 (1/30-1/31) are force set to MTU 1500 while the port channel itself is set to the default MTU value of 9216, which means that the maximum MTU is going to be the lower of the two.If you have not upgraded to 3.9.2110 then this issue can be prevented by making the following changes before upgrading.If you are currently running ONYX 3.9.2110 and experience an issue where the two switches in the MLAG pair is stuck in a peering state, then you will need to do the following to resolve the issue.If you are not comfortable performing this recommended action for the production environment, then HPE recommends that you contact HPE Customer Support and reference this Customer Advisory.The first step is to identify the master switch in the MLAG configuration, which is the switch with a higher IPL port-channel IP address (1.1.1.2 in this case):SN3700C-ONYX-ODD1 [vip-4243: master](config) # show mlagThe switch with the lower IPL port-channel IP address (in this case the IP address is 1.1.1.1) is the "standby" switch and should be modified first. Please note that DHCI configurations should have their existing network uplink ports on the standby switch disabled first to prevent any loops before moving forward with the following set of commands.Disable the IPL ports between the two switches (1/30 and 1/31 in this case) from the standby switch:interface ethernet 1/30-1/31 shutdownNext remove the IPL ports from the port channel:no interface ethernet 1/30-1/31 channel-groupChange the MTU of the IPL ports to be 9216:interface ethernet 1/30-1/31 mtu 9216If your existing port channel MTU is not set to 9216 then you can do that as well:interface port-channel 1 mtu 9216Ensure that your VLAN interface is also set to 9216:interface vlan 3000 mtu 9216Finally add the ports back to the IPL:interface ethernet 1/30-1/31 channel-group 1 mode activeinterface port-channel 1 ipl 1Repeat the above commands on the master switch. Once the IPL ports, the IPL port channels, and VLAN interfaces are confirmed to be set to 9216 by the commands below, then you can re-enable the IPL ports on the standby switch and write the changes to memory on both switches.show interface ethernet 1/30-1/31show interfaces port-channel 1show interfaces vlan 3000interface ethernet 1/30-1/31 no shutdownwrite memoryIssue ashow mlagcommand on both switches to ensure that the members are in an "UP" state:NOTE: If the MLAG status does not display "UP," reboot one switch at a time and check the MLAG status again.RECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL:Proactive Updates Subscription Form.
Operating Systems Affected:Not Applicable
Click on a version to see all relevant bugs
Hewlett Packard Enterprise Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.