Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
- Customer performed the XE upgrade in one of theirs ASR-903/RSP3, and one card A900-IMA8Z went into "missing" state: Slot Type State Insert time (ago) --------- ------------------- --------------------- ----------------- 0/0 A900-IMA8Z ok 06:41:58 0/1 A900-IMA8Z missing 06:37:14 <<<<<<<<<<<<<<<<< 0/2 A900-IMA8S ok 06:46:39 Issue may be seen also on RSP3 SO and IMs may not appear at all in the "show platform" output.
During FPD upgrade of 8x10G (IMA8Z) to 0.21/0.22 version through ISSU or router reload or RSP3 SO.
IM should recover on its own after 24 hours. Using following config command to power off and on the IM if it gets into missing state for long. hw-module subslot shutdown unpowered & no hw-module subslot shutdown unpowered If action above doesn't help, router reload will recover the problem.
- From the logs the problem seems associated with the FPD image. There is a failure on IPC communication. ------ [SNIP] *Oct 2 02:44:17.101 WET: %SYS-5-RESTART: System restarted -- Cisco IOS Software [Fuji], ASR900 Software (PPC_LINUX_IOSD-UNIVERSALK9_NPE-M), Version 16.7.1, CUST-SPECIAL:V167_1_CSCVF72306_3 This software is supported for a limited time under special agreement with Cisco Systems, Inc. CSCvf72306_3 Copyright (c) 1986-2018 by Cisco Systems, Inc. Compiled Tue 12-Jun-18 10:26 by mcpre *Oct 2 02:44:17.222 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8Z) offline in subslot 0/0 *Oct 2 02:44:17.229 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8Z) offline in subslot 0/1 *Oct 2 02:44:19.974 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8S) offline in subslot 0/2 *Oct 2 02:44:19.981 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8S) offline in subslot 0/4 *Oct 2 02:44:19.986 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8T) offline in subslot 0/5 *Oct 2 02:44:19.994 WET: %IOSXE_OIR-6-INSCARD: Card (rp) inserted in slot R1 *Oct 2 02:44:19.995 WET: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F0 *Oct 2 02:44:19.995 WET: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F0 *Oct 2 02:44:19.996 WET: %IOSXE_OIR-6-INSCARD: Card (fp) inserted in slot F1 *Oct 2 02:44:19.996 WET: %IOSXE_OIR-6-ONLINECARD: Card (fp) online in slot F1 *Oct 2 02:44:20.000 WET: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 0 *Oct 2 02:44:20.000 WET: %IOSXE_OIR-6-ONLINECARD: Card (cc) online in slot 0 *Oct 2 02:44:20.006 WET: %IOSXE_OIR-6-INSCARD: Card (cc) inserted in slot 1 *Oct 2 02:44:20.117 WET: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/0 *Oct 2 02:44:20.128 WET: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/1 *Oct 2 02:44:20.131 WET: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/2 *Oct 2 02:44:20.135 WET: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/4 *Oct 2 02:44:20.141 WET: %IOSXE_OIR-6-INSSPA: SPA inserted in subslot 0/5 *Oct 2 02:48:50.183 WET: %FPD_MGMT-5-UPGRADE_ATTEMPT: Attempting to automatically upgrade the FPD image(s) for A900-IMA8Z card in subslot 0/1. Use 'show upgrade fpd progress' command to view the upgrade progress ... *Oct 2 02:48:50.436 WET: %FPD_MGMT-6-BUNDLE_DOWNLOAD: Downloading FPD image bundle for A900-IMA8Z card in subslot 0/1 ... *Oct 2 02:48:50.437 WET: %FPD_MGMT-3-PKG_VER_MISMATCH_NOTE: The FPD image package being used (fpd:0/1/asr900-fpd-bundle.pkg) is not the right version for this IOS version (it appears that a 'asr900_rsp3-fpd-bundle.pkg' package was renamed to 'asr900-fpd-bundle.pkg'). An attempt to find the required FPD image will still be performed with this package. *Oct 2 02:48:52.940 WET: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for A900-IMA8Z card in subslot 0/1 = 00:08:00. *Oct 2 02:48:53.012 WET: %FPD_MGMT-6-UPGRADE_START: UEA GEIM 8x10G FPGA (FPD ID=37) image upgrade in progress for A900-IMA8Z card in subslot 0/1. Updating to version 0.21. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:08:00) ... *Oct 2 02:48:53.020 WET: %FPD_MGMT-3-SEND_IMG_FAILED: UEA GEIM 8x10G FPGA (FPD ID=37) image for A900-IMA8Z card in subslot 0/1 has failed to be sent for upgrade operation - Cannot find image download entry. *Oct 2 02:48:53.020 WET: %FPD_MGMT-6-OVERALL_UPGRADE: All the attempts to upgrade the required FPD images have been completed for A900-IMA8Z card in subslot 0/1. Number of successful/failure upgrade(s): 0/1. *Oct 2 02:48:53.020 WET: %FPD_MGMT-6-UPGRADE_RETRY: Attempting to recover from the failed upgrades ... *Oct 2 02:48:58.035 WET: %SPA_OIR-3-RECOVERY_RELOAD: subslot 0/1: Attempting recovery by reloading SPA *Oct 2 02:48:58.043 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8Z) offline in subslot 0/1 *Oct 2 02:49:03.929 WET: %IOSXE_OIR-6-REMSPA: SPA removed from subslot 0/1, interfaces disabled *Oct 2 02:49:03.973 WET: %SPA_OIR-6-OFFLINECARD: SPA (A900-IMA8Z) offline in subslot 0/1 *Oct 2 02:49:04.014 WET: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/0/0, changed state to down *Oct 2 02:49:04.023 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL subslot 0/0 Active Card Removed OIR Alarm *Oct 2 02:49:04.025 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT CRITICAL subslot 0/1 Active Card Removed OIR Alarm *Oct 2 02:49:37.240 WET: %FPD_MGMT-3-INCOMP_IMG_VER: Incompatible UEA GEIM 8x10G FPGA (FPD ID=37) image version detected for A900-IMA8Z card in subslot 0/1. Detected version = 0.17, minimum required version = 0.21. Current HW version = 1.0. *Oct 2 02:49:37.303 WET: %FPD_MGMT-5-UPGRADE_ATTEMPT: Attempting to automatically upgrade the FPD image(s) for A900-IMA8Z card in subslot 0/1. Use 'show upgrade fpd progress' command to view the upgrade progress ... *Oct 2 02:49:38.396 WET: %FPD_MGMT-6-BUNDLE_DOWNLOAD: Downloading FPD image bundle for A900-IMA8Z card in subslot 0/1 ... *Oct 2 02:49:38.398 WET: %FPD_MGMT-3-PKG_VER_MISMATCH_NOTE: The FPD image package being used (fpd:0/1/asr900-fpd-bundle.pkg) is not the right version for this IOS version (it appears that a 'asr900_rsp3-fpd-bundle.pkg' package was renamed to 'asr900-fpd-bundle.pkg'). An attempt to find the required FPD image will still be performed with this package. *Oct 2 02:49:40.698 WET: %FPD_MGMT-6-UPGRADE_TIME: Estimated total FPD image upgrade time for A900-IMA8Z card in subslot 0/1 = 00:08:00. *Oct 2 02:49:40.769 WET: %FPD_MGMT-6-UPGRADE_START: UEA GEIM 8x10G FPGA (FPD ID=37) image upgrade in progress for A900-IMA8Z card in subslot 0/1. Updating to version 0.21. PLEASE DO NOT INTERRUPT DURING THE UPGRADE PROCESS (estimated upgrade completion time = 00:08:00) [SNIP] ------ - However, more than 1 day the card recovered by itself: ------ ==== ====================== ====== ============================================= H/W Field Programmable Current Min. Required Slot Card Type Ver. Device: "ID-Name" Version Version ==== ====================== ====== ================== =========== ============== 0/0 A900-IMA8Z 1.0 37-UEA GEIM 8x10G 0.21 0.21 ---- ---------------------- ------ ------------------ ----------- -------------- 0/1 A900-IMA8Z 1.0 37-UEA GEIM 8x10G 0.21 0.21 <<<<<<<<<<<<<<<< ---- ---------------------- ------ ------------------ ----------- -------------- 0/2 A900-IMA8S 2.0 14-UEA GEIM I/O FP 0.49 0.47 ---- ---------------------- ------ ------------------ ----------- -------------- 0/4 A900-IMA8S 2.0 14-UEA GEIM I/O FP 0.49 0.47 ---- ---------------------- ------ ------------------ ----------- -------------- 0/5 A900-IMA8T 2.0 14-UEA GEIM I/O FP 0.49 0.47 ==== ====================== ====== ============================================= ------ - From the logs, seems that due to a timeout, the router forced a reboot to the card and it was enough to bring the card up: ------ [SNIP] Oct 3 11:02:23.549 WET: %SPA_OIR-3-EVENT_TIMEOUT: subslot 0/1: Timeout waiting for SPA OIR event -Traceback= 1#0f773eac86e9eac01f948ff1c9b61cd6 :10000000+13520F4 :10000000+48146B8 :10000000+47F7C38 :10000000+39DD950 :10000000+47FAB44 :10000000+4815F04 :10000000+3F9E09C :10000000+3F9F1CC :10000000+30D1348 Oct 3 11:02:23.689 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL subslot 0/1 Active Card Removed OIR Alarm Oct 3 11:02:28.548 WET: %SPA_OIR-3-RECOVERY_RELOAD: subslot 0/1: Attempting recovery by reloading SPA Oct 3 11:02:28.553 WET: %SYS-2-LINKED: Bad enqueue of 3B9DBCFC in queue 3B9DB934 -Process= "CWAN OIR Handler", ipl= 0, pid= 170 -Traceback= 1#0f773eac86e9eac01f948ff1c9b61cd6 :10000000+13520F4 :10000000+310D05C :10000000+47F8244 :10000000+39DD860 :10000000+47FAB44 :10000000+3F9F218 :10000000+30D1348 Oct 3 11:02:28.569 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT MAJOR IM subslot 0/1 Boot state Oct 3 11:02:36.771 WET: %IOSXE-4-PLATFORM: R1/0: kernel: irq: no irq domain found for /pcie@ffb250000/pcie@0 ! Oct 3 11:02:36.765 WET: %IOSXE-4-PLATFORM: R0/0: kernel: irq: no irq domain found for /pcie@ffb250000/pcie@0 ! Oct 3 11:02:39.347 WET: %IOSXE-4-PLATFORM: R1/0: kernel: irq: no irq domain found for /pcie@ffb250000/pcie@0 ! Oct 3 11:02:39.349 WET: %IOSXE-4-PLATFORM: R0/0: kernel: irq: no irq domain found for /pcie@ffb250000/pcie@0 ! Oct 3 11:02:56.014 WET: %SPA_OIR-6-ONLINECARD: SPA (A900-IMA8Z) online in subslot 0/1 Oct 3 11:02:58.010 WET: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/1/7, changed state to down Oct 3 11:02:59.950 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR MAJOR IM subslot 0/1 Boot state Oct 3 11:02:59.951 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/0 Transceiver Missing Oct 3 11:02:59.951 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/1 Transceiver Missing Oct 3 11:02:59.951 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/2 Transceiver Missing Oct 3 11:02:59.952 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/3 Transceiver Missing Oct 3 11:02:59.952 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/4 Transceiver Missing Oct 3 11:02:59.952 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT INFO xcvr container 0/1/5 Transceiver Missing Oct 3 11:02:59.953 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT CRITICAL xcvr container 0/1/6 Transceiver Missing - Link Down Oct 3 11:02:59.953 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT CRITICAL xcvr container 0/1/7 Transceiver Missing - Link Down Oct 3 11:03:00.380 WET: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/1/6, changed state to down Oct 3 11:03:00.031 WET: %TRANSCEIVER-6-INSERTED: R0/0: iomd: transceiver module inserted in TenGigabitEthernet0/1/6 Oct 3 11:03:00.032 WET: %TRANSCEIVER-6-INSERTED: R0/0: iomd: transceiver module inserted in TenGigabitEthernet0/1/7 Oct 3 11:03:06.388 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL xcvr container 0/1/6 Transceiver Missing - Link Down Oct 3 11:03:06.389 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT CRITICAL TenGigabitEthernet0/1/6 Physical Port Link Down Oct 3 11:03:06.717 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL xcvr container 0/1/7 Transceiver Missing - Link Down Oct 3 11:03:06.718 WET: %IOSXE_RP_ALARM-6-INFO: ASSERT CRITICAL TenGigabitEthernet0/1/7 Physical Port Link Down Oct 3 11:03:10.092 WET: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/1/6, changed state to up Oct 3 11:03:10.093 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL TenGigabitEthernet0/1/6 Physical Port Link Down Oct 3 11:03:10.098 WET: TenGigabitEthernet0/1/6 added as member-2 to port-channel4 Oct 3 11:03:10.796 WET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/1/6, changed state to up Oct 3 11:03:11.097 WET: %LINK-3-UPDOWN: Interface TenGigabitEthernet0/1/7, changed state to up Oct 3 11:03:11.097 WET: %IOSXE_RP_ALARM-6-INFO: CLEAR CRITICAL TenGigabitEthernet0/1/7 Physical Port Link Down Oct 3 11:03:11.103 WET: TenGigabitEthernet0/1/7 added as member-2 to port-channel5 Oct 3 11:03:11.797 WET: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet0/1/7, changed state to up [SNIP] Chassis type: ASR-903 Slot Type State Insert time (ago) --------- ------------------- --------------------- ----------------- 0/0 A900-IMA8Z ok 2d11h 0/1 A900-IMA8Z ok 1d03h <<<<<<<<<<<<<<<<<< 0/2 A900-IMA8S ok 2d11h 0/4 A900-IMA8S ok 2d11h