General

Due to incorrect IDPROM programming for temperature sensors, Critical CMAN process may crash due to memory corruption.

Symptom

C9410R crashing randomly at RP cmand process with core files generated. Switch> *Apr 2 20:47:30.218: %PMAN-3-RELOAD_SYSTEM: R0/0: root: Reloading: Chassis will be reloaded *Apr 2 20:47:30.696: %PMAN-3-PROCHOLDDOWN: R0/0: root: The process cmand has been helddown (rc 134) *Apr 2 20:47:30.748: %PMAN-0-PROCFAILCRIT: R0/0: pvp: A critical process cmand has failed (rc 134) *Apr 2 20:47:30.845: %PMAN-3-RELOAD_SYSTEM: R0/0: pvp: Reloading: Other RP is not standby ready. System will be reloaded *Apr 2 20:47:31.972: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet6/0/1, changed state to down *Apr 2 20:47:32.002: %LINEPROTO-5-UPDOWN: Line protocol on Interface TenGigabitEthernet1/0/1, changed state to down *Apr 2 20:47:31.946: %TRANSCEIVER-6-REMOVED: R0/0: iomd: Transceiver module removed from TenGigabitEthernet6/0/1 *Apr 2 20:47:32.972: %LINK-3-UPDOWN: Interface TenGigabitEthernet6/0/1, changed state to down *Apr 2 20:47:33.002: %LINK-3-UPDOWN: Interface TenGigabitEthernet1/0/1, changed state to downApr 2 20:47:41.881: %PMAN-5-EXITACTION: R0/0: pvp: Process manager is exiting: Critical process cmand fault on rp_0_0 (rc=134) Apr 2 20:47:52.574: %PMAN-3-PROCESS_NOTIFICATION: R0/0: pvp: System report /crashinfo/system-report_RP_0_20200402-204742-UTC.tar.gz (size: 20265 KB) generated 'show version' gives Reload Reason: CPU Usage due to Memory Pressure exceeds threshold Another reload reason is CP_RESET_CPU_GOT_RESET C9410#show logging onboard rp active uptime detail -------------------------------------------------------------------------------- UPTIME SUMMARY INFORMATION -------------------------------------------------------------------------------- First customer power on : 08/20/2019 15:24:50 Total uptime : 0 years 15 weeks 5 days 13 hours 50 minutes Total downtime : 0 years 8 weeks 0 days 6 hours 16 minutes Number of resets : 21 Number of slot changes : 2 Current reset reason : CP_RESET_CPU_GOT_RESET Current reset timestamp : 01/31/2020 13:32:27 Current slot : 5 Chassis type : 31 Current uptime : 0 years 0 weeks 2 days 21 hours 0 minutes -------------------------------------------------------------------------------- -------------------------------------------------------------------------------- UPTIME CONTINUOUS INFORMATION -------------------------------------------------------------------------------- Time Stamp | Reset | Uptime MM/DD/YYYY HH:MM:SS | Reason | years weeks days hours minutes -------------------------------------------------------------------------------- 08/20/2019 15:24:50 CP_RESET_POWER_ON 0 0 0 0 0 08/20/2019 15:30:36 CP_RESET_CPU_GOT_RESET 0 0 0 0 0 <<<<<<<<<<<<<<<<<<<<<< 08/20/2019 18:11:32 CP_RESET_POWER_ON 0 0 0 2 5 08/20/2019 18:43:17 CP_RESET_POWER_ON 0 0 0 0 5 08/20/2019 18:54:27 CP_RESET_POWER_ON 0 0 0 0 0 09/02/2019 21:37:10 CP_RESET_POWER_ON 0 0 0 0 0 09/02/2019 21:49:10 CP_RESET_CPU_GOT_RESET 0 0 0 0 5 09/03/2019 05:03:35 CP_RESET_CPU_GOT_RESET 0 0 0 0 5 09/03/2019 05:13:01 CP_RESET_CPU_GOT_RESET 0 0 0 0 5 09/03/2019 05:19:39 CP_RESET_POWER_ON 0 0 0 0 0 10/02/2019 16:10:21 CP_RESET_POWER_ON 0 0 0 0 0 10/02/2019 17:11:30 CP_RESET_CPU_GOT_RESET 0 0 0 0 5 12/02/2019 16:36:00 CP_RESET_CPU_GOT_RESET 0 8 5 0 5 12/02/2019 16:41:33 CP_RESET_CPU_GOT_RESET 0 0 0 0 0 12/11/2019 15:03:54 CP_RESET_POWER_ON 0 0 2 18 0 12/31/2019 11:37:44 CP_RESET_POWER_ON 0 1 5 16 0 01/13/2020 09:20:38 CP_RESET_POWER_ON 0 1 5 21 0 01/13/2020 09:59:08 CP_RESET_CPU_GOT_RESET 0 0 0 0 30 01/23/2020 21:20:47 CP_RESET_CPU_GOT_RESET 0 1 3 11 0 01/29/2020 16:44:26 CP_RESET_POWER_ON 0 0 5 19 0 01/31/2020 13:32:27 CP_RESET_CPU_GOT_RESET 0 0 1 20 0

Conditions

Incorrect IDPROM programming for temperature sensors

Workaround

Disable OBFL temperature logging via CLI "no hw-module slot 3 logging onboard temperature"

Further Problem Description