Symptom

On Nexus 3000 switch, NVRAM corruption could occur when switch is either upgraded to newer NX-OS release of 7.0(3)I4(x) or 7.0(3)I7(x) or if its reloaded or power cycled.

Conditions

The following messages will be seen in the log: The switch is seeing a kernel panic referencing functions related to NVRAM. Eg:

[ 9.763286] [1264723925] BUG: unable to handle kernel paging request at 00000001b1c2d3f4

[ 9.859749] [1264723925] IP: [<ffffffffa05c84ea>] ks_nvram_memcpy+0xaa/0x370 [klm_nvram]

In addition, we see the following log saying the NVRAM has a bad magic value (ie: it's corrupt):

[ 75.195634] [1264723991] klm_obfl: gave up waiting for init of module klm_nvram.

[ 75.283774] [1264723991] klm_obfl: Unknown symbol nvram_set_rr (err -16) insmod: error inserting '/isanboot/lib/modules/k[ 75.410490]

[1264723991] mtdoops: bad record bad magic 0

Workaround

=== Option 1 (Higher odds of success, but config will be erased) === 1) Power cycle the switch to force it to reload. After the text "booting kickstart image" is seen, git "CTRL+C" to get to the "switch(boot)#" prompt 2) From the "switch(boot)#" prompt, erase the NVRAM as follows. Note: This will destroy the switch's startup configuration, so please ensure a backup of all configurations is available. switch(boot)# wr erase Warning: This command will erase the startup-configuration. Do you wish to proceed anyway? (y/n) [N] y === Option 2 (Lower odds of success, but config might be left intact === 1) At bootup, hit "CTRL+L" immediately and repeatedly to get to the switch loader> prompt 2) Issue "boot (name of NX-OS image)". Note: In some cases, booting a different or older NX-OS image might let the switch boot into NX-OS, although still with NVRAM errors present. 3) If a different NX-OS image is able to successfully load, issue a "wr erase" and then "copy run start". This will re-initialize the NVRAM.

Further Problem Description