OPERATIONAL DEFECT DATABASE
...
...
When upgrading a Nexus 9000 switch from a code where LXC boot mode is not the default mode, to one where it is, using non-disruptive ISSU, the "radius-server key 7 " configuration can go missing. This is due to to the change in default boot mode.
Upgrade from earlier N9K release on applicable platforms to later code (10.3.3+ or 10.4.1+) where LXC boot mode is the default. To confirm the boot mode, run "show boot mode", or check for VSUP for TOR switch in the "show module" output.
Remove and reapply the missing CLI string to reconfigure and ensure it is consistent between configuration and DME.
There is another instance of radius/tacacs keys lost on upgrade within Native mode. Here keys are stored in software TAM before and after the upgrade. Here the keys are lost or wrong key values are read in the running-config from TAM, because of the stale TAM directory present in the bootflash, here also DME will have proper key values restored. One way this stale TAM directory can come is due to cold reboot (copy r s & reload) instead of doing ISSU in the history of upgrades performed in the box, in earlier releases different versions of TAM coexist and ISSU will make sure only one TAM directory is present at a time. Stale TAM directories can lead to undefined behaviours in reading the keys. The workarounds for recovering the issue are discussed below. 1. Remove the TAM directory as a whole, and do a reload and configure the keys again. This time stale TAM directory that got carried over from previous history of upgrades will get deleted, and doing a proper ISSU further ensure only one TAM directory is present in further upgrades , thus preventing the issue. Steps are given below :- a. Run bash -> sudo rm -rf /bootflash/.swtam b. reload c. reconfigure radius key d. copy r s e. Reload 2. Another workaround is to configure keys on a per server basis instead of configuring it globally for radius/tacacs. Per server keys are not stored in TAM, thus the issue can be prevented. For instance , the configuration would be like :- tacacs-server host key 7 "******" This issue will be hit only in releases before 10.4.2 , because even though the fix of this bug is present in the releases such as 10.2.x and 10.3.x, due to stale TAM directory the fix is not taking into effect . Upgrading to releases above 10.4.2 prevent the user from hitting this keys getting lost issue during upgrade also, since the fix of this bug will take into effect properly because the TAM is disabled in releases above 10.4.2. This will ensure the issue not getting hit in releases above 10.4.2
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
