Symptom

The following log messages are seen: 2022 Aug 30 09:24:56 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %SYSMGR-SLOT25-2-SERVICE_CRASHED: Service "xbar_client" (PID 24504) hasn't caught signal 6 (no core). 2022 Aug 30 10:34:58 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %SYSMGR-SLOT22-2-SERVICE_CRASHED: Service "xbar_client" (PID 403) hasn't caught signal 6 (no core). 2022 Aug 30 10:34:59 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %SYSMGR-SLOT22-2-HAP_FAILURE_SUP_RESET: Service "xbar_client" in vdc 1 has had a hap failure During the investigation, we noticed communication errors between the LineCards and Supervisor after the Service crashes: 2022 Aug 30 10:35:01 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %KERN-0-SYSTEM_MSG: [2983201.382416] EMON: module 22 is not responding on EOBC path. Reloading module. - kernel 2022 Aug 30 10:35:01 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 22 (Serial number: JAE25110E3A) reported failure due to EOBC heartbeat failure in device DEV_EOBC_MAC (device error 0xc0a1b137) 2022 Aug 30 10:35:21 CORE-A-MGMT-1GMT1 %$ VDC-1 %$ %MODULE-2-MOD_DIAG_FAIL: Module 22 (Serial number: JAE25110E3A) reported failure due to Service on linecard had a hap-reset in device DEV_SYSMGR (device error 0x49)

Conditions

This condition was identified on a N9K-C9508m running NX-OS nxos64-msll.10.2.3.F.bin. Also, it was identified multiple core file from the same service in different Fabric modules: `show cores` VDC Module Instance Process-name PID Date(Year-Month-Day Time) --- ------ -------- --------------- -------- ------------------------- 1 21 1 xbar_client 15520 2022-08-26 17:54:38 1 24 1 xbar_client 15512 2022-08-26 17:56:49 1 25 1 xbar_client 15512 2022-08-26 18:38:43 1 22 1 xbar_client 15513 2022-08-26 18:39:17 1 23 1 xbar_client 15521 2022-08-26 18:39:35 1 26 1 xbar_client 15515 2022-08-26 18:41:03 1 23 1 xbar_client 15510 2022-08-26 21:24:38 1 23 1 xbar_client 15508 2022-08-27 04:20:01 1 21 1 xbar_client 7564 2022-08-28 17:57:24 1 26 1 xbar_client 9063 2022-08-28 18:28:28 1 23 1 xbar_client 27991 2022-08-28 18:48:44 1 22 1 xbar_client 8946 2022-08-28 18:50:01 1 25 1 xbar_client 8974 2022-08-28 19:02:57 1 24 1 xbar_client 7655 2022-08-28 19:04:56 1 25 1 xbar_client 15511 2022-08-29 00:03:36 1 24 1 xbar_client 15510 2022-08-29 00:20:32 1 21 1 xbar_client 31362 2022-08-30 08:43:24 1 25 1 xbar_client 24504 2022-08-30 09:27:48 1 22 1 xbar_client 403 2022-08-30 10:35:21 1 23 1 xbar_client 1743 2022-08-30 10:57:17 `show module` Mod Ports Module-Type Model Status --- ----- ------------------------------------- --------------------- --------- snip... 21 0 Fabric Module N9K-C9508-FM-R2 ok 22 0 Fabric Module N9K-C9508-FM-R2 ok 23 0 Fabric Module N9K-C9508-FM-R2 ok 24 0 Fabric Module N9K-C9508-FM-R2 ok 25 0 Fabric Module N9K-C9508-FM-R2 ok 26 0 Fabric Module N9K-C9508-FM-R2 ok

Workaround

A full reload of the system has been performed. No xbar_client has seen since then.

Further Problem Description