Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Nexus 9000/3000 switch bootflash goes into read-only mode with M500IT SSD drive after 28,224 power-on-hours (POH) for the first time. The bootflash will stop responding causing failure of operations such as config changes/save, read/write operations etc. syslogs will also indicate bootflash diagnostic test failure %$ VDC-1 %$ %DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test has been disabled as a part of default EEM action %$ VDC-1 %$ %DEVICE_TEST-2-COMPACT_FLASH_FAIL: Module 1 has failed test BootFlash 5 times on device BootFlash due to error Failure
This cli can be used for any NXOS version switch# conf t switch(config)# feature bash switch(config)# run bash sudo su bash-4.2# smartctl -a /dev/sda | egrep 'Model|Firmware|Hours' Device Model: Micron_M500IT_MTFDDAT064SBD Firmware Version: MU01.00 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 4872 bash-4.2# Affected Switch will have both of the following conditions True from the above output 1) Model starts with Micron_M500IT 2) FW Version is either MU01.00 or MC02.00 Power_On_Hours from the above output can be used to calculate how much time is left to hit this issue. For eg: In the above output the current PoH (power-on-hours) is 4872. This issue may be seen on following PIDs: N9K-C9236C, N9K-C9396TX, N9K-C9396PX, N9K-C93128TX, N9K-C9332PQ, N9K-C9372PX, N9K-C9372TX, N9K-C93120TX, N9K-C9372PX-E, N9K-C9372TX-E, N9K-C92160YC-X, N9K-C9272Q, N9K-C93180YC-EX, N9K-C93108TC-EX, N9K-C92304QC, N9K-C9232C, N9K-C93180YC-EX-24, N9K-C93108TC-EX-24, N9K-C93180LC-EX, N9K-SUP-B+, N9K-SUP-B, N9K-SUP-A+, N9K-SUP-A, N3K-C3232C, N3K-C3264Q, N3K-C31128PQ-10GE, N3K-C31108PC-V, N3K-C31108TC-V, N3K-C3164Q-40GE
Reload the switch. However, this failure will reappear after 1008 hours of operation. Important Pre-check before deciding on upgrade options: 1. Verify if the bootflash is already in the failed state (read-only state). Then, power-cycle/reload the switch in order to recover the bootflash. 2. Check to see if the Temperature_Celsius attribute is 128 with the commajd: smartctl -a /dev/sda | egrep 'Temperature_Celsius|ID#' If it is 128, then power-cycle/reload the switch before you proceed with the SSD Firmware upgrade options. Note: An upgrade of the SSD Firmware of the switch with a RAW_VALUE of 128 might result in unexpected behavior after a firmware upgrade (for example, an unexpected reload or read-only drive). Any RAW_VALUE other than 128 for Temperature_Celsius is valid. Configure bash if not enabled and then run bash: switch# feature bash switch# run bash sudo su bash-4.2# For Nexus 9500, enter the rlogin command from the Active supervisor in order to log in to Standby supervisor. If slot 28 is the Standby supervisor, enter this command: bash-4.2# rlogin sup28 root@switch# If slot 27 is the Standby supervisor, enter this command: bash-4.2# rlogin sup27 root@switch# bash-4.4# smartctl -a /dev/sda | egrep 'Temperature_Celsius|ID#' ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 128 (x xx x xx xxx) nxos.CSCvx21260-n9k_ALL-1.0.1-.lib32_n9000.rpm bundle (Note: 1.0.1) automatically performs this Temperature_Celsius attribute precheck. If the Temperature_Celsius attribute is read as 128, it will bail out and recommend a reload of the switch to the user. Upgrade the SSD Firmware using following options • Option 1 - Upgrade NXOS Version The new Firmware with the fix for this issue will be packaged in 9.3(7), 10.1(2) and later NXOS versions. • Option 2 - Upgrade SSD FW using SMU This option will be available under NX-OS Software Maintenance Upgrades (SMU) for 7.0(3)I7(x), 9.2(x), 9.3(x) and 10.1(1) on affected PIDs • 7.0(3)I7(9) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-7.0.3.I7.9.lib32_n9000.rpm is applicable to NX-OS Software Release 7.0(3)I7(1) to 7.0(3)I7(9) • 9.3(6) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-9.3.6.lib32_n9000.rpm is applicable to NX-OS Software Release 9.2(1) - 9.2(4) and 9.3(1) - 9.3(6) • 10.1(1) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-10.1.1.lib32_n9000.rpm is applicable to NX-OS Software Release 10.1(1) • Option 3 - Upgrade SSD FW using script Script upgrade_m500_firmware.tar.gz will be available under NX-OS Firmware section for the affected PIDs. This same script can be used for any version from 7.0(3)I7(x), 9.2(x), 9.3(x), 10.1(1) 1. Copy upgrade_m500_firmware.tar.gz to switch bootflash. For 9500 Series Switches with Dual Supervisor, copy upgrade_m500_firmware.tar.gz to active as well as standby supervisor bootflash. Perform the upgrade first on standby supervisor and then active supervisor 2. Verify that upgrade_m500_firmware.tar.gz is in bootflash switch# dir bootflash: | grep upgrade 2151467 Mar 08 19:17:00 2021 upgrade_m500_firmware.tar.gz In case of Nexus 9500, verify upgrade_m500_firmware.tar.gz is also in Standby Supervisor bootflash switch# dir bootflash://sup-standby/ | grep upgrade 2151467 Mar 08 19:18:00 2021 upgrade_m500_firmware.tar.gz 3. Configure bash if not enabled and run bash switch# feature bash switch# run bash sudo su bash-4.2# In case of Nexus 9500, login to standby supervisor for Nexus 9500 use rlogin command from active supervisor If slot 28 is Standby supervisor, then bash-4.2# rlogin sup28 root@switch# If slot 27 is Standby supervisor, then bash-4.2# rlogin sup27 root@switch# 4. Copy the script from bootflash to /tmp bash-4.2# cp /bootflash/upgrade_m500_firmware.tar.gz /tmp 5. Uncompress the file in /tmp folder bash-4.2# cd /tmp bash-4.2# tar -xvzf upgrade_m500_firmware.tar.gz upgrade_m500_firmware M500_MC03.bin M500_MU05.bin 6. IMPORTANT - Execute the script upgrade_m500_firmware with no parameters bash-4.2# ./upgrade_m500_firmware Checking SSD firmware ... Model Number: Micron_M500IT_MTFDDAT064SBD Serial Number: ############## Firmware Revision: MU01.00 SSD Model: Micron_M500IT_MT Current SSD Firmware Version: 1 Your SSD firmware needs update and will be upgraded Updating the SSD firmware ... /dev/sda: fwdownload: xfer_mode=3 min=1 max=255 size=512 ............................................................................... Done. Model Number: Micron_M500IT_MTFDDAT064SBD Serial Number: MSA2226001B Firmware Revision: MU05.00 Current SSD Firmware is 5 SSD Firmware has been updated successfully Please Note: After Upgrade SSD Firmware will either be MU05.00 or MC03.00.
https://www.cisco.com/c/en/us/support/docs/field-notices/721/fn72145.html