Symptoms

The specific port range goes into "hw_Failure" state: `show interface brief` ----------------------------------------------------------------------------------------- Interface Vsan Admin Admin Status SFP Oper Oper Port Logical Mode Trunk Mode Speed Channel Type Mode (Gbps) ----------------------------------------------------------------------------------------- fc9/41 1400 FX off hwFailure swl -- -- -- -- fc9/42 1400 FX off hwFailure swl -- -- -- -- fc9/43 1400 FX off hwFailure swl -- -- -- -- fc9/44 1400 FX off hwFailure swl -- -- -- -- fc9/45 1 FX off hwFailure swl -- -- -- -- fc9/46 1 FX off hwFailure swl -- -- -- -- fc9/47 1400 FX off hwFailure swl -- -- -- -- fc9/48 1 E on hwFailure swl -- -- 57 -- The error code is seen in the module internal exception log as below: `show module internal exceptionlog module 9` ********* Exception info for module 9 ******** exception information --- exception instance 1 ---- Module Slot Number: 9 Device Id : 204 Device Name : F16 Generic Driver Device Errorcode : 0xccc05600 Device ID : 204 (0xcc) Device Instance : 05 (0x05) Dev Type (HW/SW) : 06 (0x06) ErrNum (devInfo) : 00 (0x00) System Errorcode : 0x42b8001e fatal error Error Type : FATAL error PhyPortLayer : Fibre Channel Port(s) Affected : fc9/41-48 Error Description : F16_PLDA_RETRY_MERR DSAP : 0 (0x0) Time : Mon Jan 6 22:22:32 2025 (Ticks: 677CAC08 jiffies) A module reset is seen in the show logging nvram: `show logging nvram` 2025 Jan 6 22:22:32 WTXA19710H15 %MODULE-2-MOD_SOMEPORTS_FAILED: Module 9 (Serial number: JAE18280N1K) reported failure on ports fc9/41-48 (Fibre Channel) due to fatal error in device DEV_F16_CMN (device error 0xccc05600) 2025 Jan 6 22:22:33 WTXA19710H15 %CALLHOME-2-EVENT: PORT_FAILURE `show logging log` 2025 Jan 6 22:22:32 WTXA19710H15 %MODULE-2-MOD_SOMEPORTS_FAILED: Module 9 (Serial number: JAE18280N1K) reported failure on ports fc9/41-48 (Fibre Channel) due to fatal error in device DEV_F16_CMN (device error 0xccc05600) 2025 Jan 6 22:22:32 WTXA19710H15 %PORT-CHANNEL-5-PORT_DOWN: port-channel57: fc9/48 is down 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1%$ Interface fc9/48 is down (Hardware Failure) port-channel57 ISL to WTXA19710C02 fc7/22 2025 Jan 6 22:22:33 WTXA19710H15 %CALLHOME-2-EVENT: PORT_FAILURE 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1400%$ Interface fc9/47 is down (Hardware Failure) ltx15brwccas01_h0 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1%$ Interface fc9/46 is down (Hardware Failure) 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1%$ Interface fc9/45 is down (Hardware Failure) 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1400%$ Interface fc9/44 is down (Hardware Failure) 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1400%$ Interface fc9/43 is down (Hardware Failure) 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1400%$ Interface fc9/42 is down (Hardware Failure) ltx14brwccas02_h0 2025 Jan 6 22:22:33 WTXA19710H15 %PORT-5-IF_DOWN_HW_FAILURE: %$VSAN 1400%$ Interface fc9/41 is down (Hardware Failure) ltx14brwccas01_h0

Cause

The DEV_F16_CMN error on a Cisco MDS 9000 series switch typically indicates a hardware issue related to the F16 ASIC. This error often results in the module rebooting due to an unrecoverable multi-bit Error-Correcting Code (ECC) error.

Resolution

The only way to recover an interface from hw_Failure is to disruptively reload the module. #reload module x If the interfaces are in a temporary hardware failure, status is recoverable by module reload. If the interfaces have a permanent hardware failure, then proceed with module replacement. WARNING: The activity is disruptive and should be performed during the maintenance window.

Support Cases