OPERATIONAL DEFECT DATABASE
...
...
Nexus 9000/3000 switch bootflash goes into read-only mode with M500IT SSD drive after 28,224 power-on-hours (POH) for the first time. The bootflash will stop responding causing failure of operations such as config changes/save, read/write operations etc. syslogs will also indicate bootflash diagnostic test failure %$ VDC-1 %$ %DIAGCLIENT-2-EEM_ACTION_HM_SHUTDOWN: Test has been disabled as a part of default EEM action %$ VDC-1 %$ %DEVICE_TEST-2-COMPACT_FLASH_FAIL: Module 1 has failed test BootFlash 5 times on device BootFlash due to error Failure
This cli can be used for any NXOS version switch# conf t switch(config)# feature bash switch(config)# run bash sudo su bash-4.2# smartctl -a /dev/sda | egrep 'Model|Firmware|Hours' Device Model: Micron_M500IT_MTFDDAT064SBD Firmware Version: MU01.00 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 4872 bash-4.2# Affected Switch will have both of the following conditions True from the above output 1) Model starts with Micron_M500IT 2) FW Version is either MU01.00 or MC02.00 Power_On_Hours from the above output can be used to calculate how much time is left to hit this issue. For eg: In the above output the current PoH (power-on-hours) is 4872. This issue may be seen on following PIDs: N9K-C9236C, N9K-C9396TX, N9K-C9396PX, N9K-C93128TX, N9K-C9332PQ, N9K-C9372PX, N9K-C9372TX, N9K-C93120TX, N9K-C9372PX-E, N9K-C9372TX-E, N9K-C92160YC-X, N9K-C9272Q, N9K-C93180YC-EX, N9K-C93108TC-EX, N9K-C92304QC, N9K-C9232C, N9K-C93180YC-EX-24, N9K-C93108TC-EX-24, N9K-C93180LC-EX, N9K-SUP-B+, N9K-SUP-B, N9K-SUP-A+, N9K-SUP-A, N3K-C3232C, N3K-C3264Q, N3K-C31128PQ-10GE, N3K-C31108PC-V, N3K-C31108TC-V, N3K-C3164Q-40GE
Reload the switch. However, this failure will reappear after 1008 hours of operation. Important Pre-check before deciding on upgrade options: 1. Verify if the bootflash is already in the failed state (read-only state). Then, power-cycle/reload the switch in order to recover the bootflash. 2. Check to see if the Temperature_Celsius attribute is 128 with the commajd: smartctl -a /dev/sda | egrep 'Temperature_Celsius|ID#' If it is 128, then power-cycle/reload the switch before you proceed with the SSD Firmware upgrade options. Note: An upgrade of the SSD Firmware of the switch with a RAW_VALUE of 128 might result in unexpected behavior after a firmware upgrade (for example, an unexpected reload or read-only drive). Any RAW_VALUE other than 128 for Temperature_Celsius is valid. Configure bash if not enabled and then run bash: switch# feature bash switch# run bash sudo su bash-4.2# For Nexus 9500, enter the rlogin command from the Active supervisor in order to log in to Standby supervisor. If slot 28 is the Standby supervisor, enter this command: bash-4.2# rlogin sup28 root@switch# If slot 27 is the Standby supervisor, enter this command: bash-4.2# rlogin sup27 root@switch# bash-4.4# smartctl -a /dev/sda | egrep 'Temperature_Celsius|ID#' ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 194 Temperature_Celsius 0x0022 100 100 000 Old_age Always - 128 (x xx x xx xxx) nxos.CSCvx21260-n9k_ALL-1.0.1-.lib32_n9000.rpm bundle (Note: 1.0.1) automatically performs this Temperature_Celsius attribute precheck. If the Temperature_Celsius attribute is read as 128, it will bail out and recommend a reload of the switch to the user. Upgrade the SSD Firmware using following options • Option 1 - Upgrade NXOS Version The new Firmware with the fix for this issue will be packaged in 9.3(7), 10.1(2) and later NXOS versions. • Option 2 - Upgrade SSD FW using SMU This option will be available under NX-OS Software Maintenance Upgrades (SMU) for 7.0(3)I7(x), 9.2(x), 9.3(x) and 10.1(1) on affected PIDs • 7.0(3)I7(9) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-7.0.3.I7.9.lib32_n9000.rpm is applicable to NX-OS Software Release 7.0(3)I7(1) to 7.0(3)I7(9) • 9.3(6) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-9.3.6.lib32_n9000.rpm is applicable to NX-OS Software Release 9.2(1) - 9.2(4) and 9.3(1) - 9.3(6) • 10.1(1) SMU nxos.CSCvx21260-n9k_ALL-1.0.0-10.1.1.lib32_n9000.rpm is applicable to NX-OS Software Release 10.1(1) • Option 3 - Upgrade SSD FW using script Script upgrade_m500_firmware.tar.gz will be available under NX-OS Firmware section for the affected PIDs. This same script can be used for any version from 7.0(3)I7(x), 9.2(x), 9.3(x), 10.1(1) 1. Copy upgrade_m500_firmware.tar.gz to switch bootflash. For 9500 Series Switches with Dual Supervisor, copy upgrade_m500_firmware.tar.gz to active as well as standby supervisor bootflash. Perform the upgrade first on standby supervisor and then active supervisor 2. Verify that upgrade_m500_firmware.tar.gz is in bootflash switch# dir bootflash: | grep upgrade 2151467 Mar 08 19:17:00 2021 upgrade_m500_firmware.tar.gz In case of Nexus 9500, verify upgrade_m500_firmware.tar.gz is also in Standby Supervisor bootflash switch# dir bootflash://sup-standby/ | grep upgrade 2151467 Mar 08 19:18:00 2021 upgrade_m500_firmware.tar.gz 3. Configure bash if not enabled and run bash switch# feature bash switch# run bash sudo su bash-4.2# In case of Nexus 9500, login to standby supervisor for Nexus 9500 use rlogin command from active supervisor If slot 28 is Standby supervisor, then bash-4.2# rlogin sup28 root@switch# If slot 27 is Standby supervisor, then bash-4.2# rlogin sup27 root@switch# 4. Copy the script from bootflash to /tmp bash-4.2# cp /bootflash/upgrade_m500_firmware.tar.gz /tmp 5. Uncompress the file in /tmp folder bash-4.2# cd /tmp bash-4.2# tar -xvzf upgrade_m500_firmware.tar.gz upgrade_m500_firmware M500_MC03.bin M500_MU05.bin 6. IMPORTANT - Execute the script upgrade_m500_firmware with no parameters bash-4.2# ./upgrade_m500_firmware Checking SSD firmware ... Model Number: Micron_M500IT_MTFDDAT064SBD Serial Number: ############## Firmware Revision: MU01.00 SSD Model: Micron_M500IT_MT Current SSD Firmware Version: 1 Your SSD firmware needs update and will be upgraded Updating the SSD firmware ... /dev/sda: fwdownload: xfer_mode=3 min=1 max=255 size=512 ............................................................................... Done. Model Number: Micron_M500IT_MTFDDAT064SBD Serial Number: MSA2226001B Firmware Revision: MU05.00 Current SSD Firmware is 5 SSD Firmware has been updated successfully Please Note: After Upgrade SSD Firmware will either be MU05.00 or MC03.00.
https://www.cisco.com/c/en/us/support/docs/field-notices/721/fn72145.html
Click on a version to see all relevant bugs
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
