Symptoms

Upgrade to Fabric OS to FOS v7.3.1e, v7.3.2a, v7.4.1c, and v8.0.1 or later on these code streamsImpact:Switch panic and cold reboot. Environment:EMC Hardware: Connectrix DS-300BEMC Hardware: Connectrix DS-5100BEMC Hardware: Connectrix DS-5300BEMC Hardware: Connectrix DS-6505BEMC Hardware: Connectrix DS-6510BEMC Hardware: Connectrix DS-6520BEMC Hardware: Connectrix MP-7840BEMC Hardware: Connectrix ED-DCX-BEMC Hardware: Connectrix ED-DCX-4SEMC Hardware: Connectrix ED-8510-4BEMC Hardware: Connectrix ED-8510-8BBrocade Software: Fabric OS 7.2.1Brocade Software: Fabric OS 7.3.2bBrocade Software: Fabric OS 7.3.1dProblem:When upgrading a switch, in a rare condition it might panic and cold boot the switch.During the upgrade process on a director itch the newly upgraded CP panicked while the new standby CP was not fully up and in sync yet, causing a cold reboot and all the fc ports to be initialized. Errdump log:[HAM-1004], 26486, SLOT 6 | CHASSIS, INFO, itbscaa1c02, Processor rebooted - Software Fault:Kernel Panic.[TRCE-1001], 26487, SLOT 6 | CHASSIS, WARNING, itbscaa1c02, Trace dump available (Slot 6)! (reason: PANIC). The panic dump shows:.Warn: INCONSISTENT, msg blk free, type: 1 Data: 43b28818:00000004:97c30a18:00000000kernel BUG in mblk_free at ../fabos/bccb/include/sys/fabos/kmsg_pvt.h:678!Oops: Exception in kernel mode, sig: 5 [#1]NIP: C3A23E58 LR: C3A23E58 SP: B23D7C60 REGS: b23d7bb0 TRAP: 0700 DBCR0:00000000 Tainted: P[Truncated] Notice in the panic dump the Warning Inconsistent in the first line. Warn: INCONSISTENT And also notice the back port listed address 43b28818 which translated to the front end port index 367 11/88.

Cause

The panic occurred due to an inconsistency and a mismatch in the media read out of the port. The fabric OS expects the buffer of the message buffer to be free, but still the system calls for it to free the message.Normally the system pulls the media data periodically, for every online port at least every 5 minutes.During hafailover, which is a normal process during firmwaredownload, the system mishandled the message buffer reference count. All online ports will go through a validation procedure to compare internal port speed against the SFP supported speed.This is due to a mismatch in speed handling if there is a mismatch in switch internal data and any online user port media speed settings, the switch will panic when upgrade to FOS v7.3.1e, v7.3.2a, v7.4.1c, and v8.0.1 or later on these code streams.A switch Director upgrade could be a cold boot. In a rare occasion any potential hafailover can have this issue. Details about the mismatch:Observed in the past, the following are possible ways that the mismatch originated (before the next hafailover). The SFP had been swapped out in pre FOS v7.2.0 release when the port was in disabled state, such a SFP update was NOT updated properly within FOS (a mismatch condition with the actual speed).The second condition is that in a rare case like light was cut-off during or immediately after previous hafailover window(s) (software has a blind window during hafailover or firmware upgrade conditions), that will result in this kind of mismatch between internal port speed and actual SFP supported speed. The result is that a port can come up at a higher speed than what SFP should or can support (but it can negotiate to a higher speed). Then the next hafailover, it is possible to hit this issue. There is also history mentioned about a bad SFP read may result in using a default speed setting, and thus eventually a mismatch. Brocade DEFECT000632217 and DEFECT000632989

Resolution

Fix:Before upgrading to the fixed version check for any mismatch. (see Notes)Upgrade to Fabric OS 7.4.1e3, v7.4.2, v8.0.2 Workaround:Check for possible mismatch. (see Notes)Brocade DEFECT000632217 and DEFECT000632989

Support Cases