Symptoms

The active CP panicked but standby did not take over and the switch reset to recover: 2024/07/09-10:35:04 (+03), [HAM-1004], 32211, SLOT 1 | CHASSIS, INFO, CA512FC100, Processor rebooted - OCP or Silent Reboot. 2024/07/09-10:35:12 (+03), [FVS-1001], 32212, SLOT 1 | CHASSIS, INFO, A512FC100, Flow Vision Stats daemon initialized. 2024/07/09-10:35:12 (+03), [FV-1001], 32213, SLOT 1 | CHASSIS, INFO, A512FC100, Flow Vision daemon initialized. 2024/07/09-10:35:18 (+03), [ESM-1002], 32214, SLOT 1 | FID 128, INFO, A512FC100, ESMd initialization done for service fcsw:0. 2024/07/09-10:35:20 (+03), [ESM-3000], 32215, SLOT 1 | FID 128, INFO, A512FC100, Cold Recovery starting. 2024/07/09-10:35:20 (+03), [ESM-3001], 32216, SLOT 1 | FID 128, INFO, A512FC100, Cold Recovery complete. 2024/07/09-10:35:20 (+03), [TO-1005], 32217, SLOT 1 | FID 128, INFO, A512FC100, TO Profile Sys_TrafOpt_Version2 is enforced successfully. 2024/07/09-10:35:21 (+03), [FCR-1069], 32218, SLOT 1 | FID 128, INFO, A512FC100, The FC Routing service is enabled. 2024/07/09-10:35:21 (+03), [FCR-1068], 32219, SLOT 1 | FID 128, INFO, A512FC100, The FC Routing service is disabled. 2024/07/09-10:35:24 (+03), [NSM-1004], 32220, SLOT 1 | FID 128, INFO, A512FC100, Interface vlan 1 is created. 2024/07/09-10:35:24 (+03), [NSM-1004], 32221, SLOT 1 | FID 128, INFO, A512FC100, Interface vlan 1002 is created. 2024/07/09-10:35:24 (+03), [ZONE-1022], 32222, SLOT 1 | FID 128, INFO, A512FC100, The effective configuration has changed to SANDR2_CFG. 2024/07/09-10:35:24 (+03), [ESM-3001], 32223, SLOT 1 | FID 128, INFO, A512FC100, Cold Recovery complete. No CP was active for a short time then the switch reset and all the blades were reset: 07/09/2024-10:35:14,Software Fault:Kernel Panic 2024/07/09-10:35:39 (+03), [HAM-1004], 32231, SLOT 2 | CHASSIS, INFO, CA512FC100, Processor rebooted - Software Fault:Kernel Panic. 2024/07/09-10:35:39:827509 (+03), [HAM-1004], 39265/32231, SLOT 2 | CHASSIS, INFO, CA512FC100, Processor rebooted - Software Fault:Kernel Panic., reboot.c, line: 122, comp:hamd, ltime:20 24/07/09-10:35:33:214225 Kernel panic - not syncing: Watchdog Timeout Error Panic due to PANIC PANIC_DUMP_LOG... panic_dump_notifier nb=c280af5c a=0 pdcheck: info: found new pd: mtd_ts=1720510448 PANIC Tue Jul 9 10:34:08 2024, cf_ts=0

Cause

This is observed on FOS9.1.1x and above where FOS lock tracing contexts are not cleaned up properly. The issue exists in 9.1.1c, 9.1.1d, 9.2.0, 9.2.0a and 9.2.0b. The CP goes for watchdog exception due to excessive printk in the spinlock context and these debug prints will be printed when syslock infrastructure detects a lock order violation. 

Resolution

Resolution: The issue is resolved in v9.2.0c as per the below release notes:https://docs.broadcom.com/doc/FOS-920c-RN

Support Cases