Symptoms

This article explains how to recover the switch from inconsistent state while doing a downgrade of firmware version in Dell N2000 and N3000. Requirement Console access to the switch is a must. You will need physical access to the switch console port in order to complete the process. You will need a personal computer with a serial port or a USB to serial adapter and the serial to RJ45 cable that comes with the switch. Cause The switch is stuck in an inconsistent state due to incompatible CPLD version.A switch goes into an inconsistent state, when the CPLD installed is a newer version than the firmware, to be downgraded. Anytime you downgrade the firmware you must consider the CPLD version and downgrade it if needed.For example, firmware version 6.0.1.3 requires CPLD version 13.If a customer is downgrading the switch from a higher firmware version 6.6.0.2 with CPLD 20 directly to 6.0.1.3, without following the proper downgrade procedure, the switch may get stuck in an inconsistent state.You can find the CPLD version compatible for the firmware version in the release notes provided with the corresponding firmware download file. How to know if a switch is stuck in an inconsistent state While booting the switch a log message is seen mentioning the "system is restarting due to inconsistent state". Sample console logs Below is a sample console log of switch while booting which shows the switch is in an inconsistent state (Omitting complete boot sequence for brevity).Extracting Operational Code from .stk file...2done.Loading Operational Code...done.Decompressing Operational Code...done.Uncompressing apps.lzmaSyncDB Running...File: stm_api.c, Line: 62, Error 0 (0x0) May 14 01:38:40 0.0.0.0-0 General[fp_main_task]: stm_api.c(62) 1 %% Error 0 (0x0)DMA pool size: 16777216AXI unit 0: Dev 0xb342, Rev 0x01, Chip BCM56342_A0, Driver BCM56340_A0In hpcLocalUnitTypeIdGet:611, could not find the boardIn sysapiHpcStmUnitDbEntryLocalUnitIndexGet:342, could not find the boardThe system is restarting due to the inconsistent state 0 in file: stm_api.c line 62 May 14 01:38:43 0.0.0.0-1 LOG[fp_main_task]: log_cnfgr.c(1357) 3 %% The system is restarting due to inconsistent state. Can't get unit number! Switching software SIGSEGV HandlerThis build was configured to copy this crash information to a file.Symbols already loaded.starting pid 12syncing filesystems....This may take a few momentsRebooting system!The system is going down NOW!Sent SIGTERM to all processesSent SIGKILL to all processesRequesting system reboot Solution Boot the switch in correct firmware version (compatible with the CPLD version) and proceed with proper downgrade procedure. Option 1 Boot the switch in backup partition. There is a chance that customer has just copied the older firmware to backup partition and is trying to reboot switch in backup partition.Dell Networking Boot Options============================Select a menu option within 3 seconds or the Operational Code will start automatically...1 - Start Operational Code2 - Display Boot MenuWhen the boot menu appears, select 2 "Display Boot Menu".Select (1, 2) #2Boot Main Menu==============1 - Start Operational Code2 - Select Baud Rate3 - Retrieve Logs4 - Load New Operational Code5 - Display Operational Code Details9 - Reboot10 - Restore Configuration to Factory Defaults11 - Activate Backup Image12 - Start Password RecoveryFrom Boot Main Menu select 11 "Activate Backup Image".Enter Choice# 11After this, reboot the switch with option 9 "Reboot".Boot Main Menu==============1 - Start Operational Code2 - Select Baud Rate3 - Retrieve Logs4 - Load New Operational Code5 - Display Operational Code Details9 - Reboot10 - Restore Configuration to Factory Defaults11 - Activate Backup Image12 - Start Password RecoveryEnter Choice# 9Option 2If the 1st option did not work, try to load firmware using TFTP/XMODEM. Follow the procedure in upgrade guide for switch recovery. If the previous firmware version is known, upgrade to that version.If the firmware version is unknown, upgrade it to the latest firmware version available Once the switch recovers follow the correct downgrade procedure provided in the upgrade guide given with the firmware download file. Upgrade to intermediate firmware release N3000_N2000vB.6.3.2.stk and downgrade CPLD to 13.

Cause

The switch is stuck in an inconsistent state due to incompatible CPLD version.A switch goes into an inconsistent state, when the CPLD installed is a newer version than the firmware, to be downgraded. Anytime you downgrade the firmware you must consider the CPLD version and downgrade it if needed.

Resolution

Option 1 Boot the switch in backup partition. There is a chance that customer has just copied the older firmware to backup partition and is trying to reboot switch in backup partition.Option 2If the 1st option did not work, try to load firmware using TFTP/XMODEM. Follow the procedure in upgrade guide for switch recovery.

Support Cases