Symptom

Switch may become completely unresponsive and fully isolated * only chassis LED lit is STATUS light * console and management interface are unresponsive * control-plane stops responding * interfaces are all down On more recent versions of code a hap reset will be detected, which causes the switch to self-reload and become operational again. On older versions of code when there is no hap-reset, the only way to recover is to manually power cycle the switch * Syslogs may show the following process crash: ascii-cfg * Switch may experience hap-resets for tahusd, l2fm, vdc_mgr or pltfm_config sh logg nvram | in core 2021 Dec 11 05:40:38.782532 LEAF47 %$ VDC-1 %$ %SYSMGR-2-SERVICE_CRASHED: Service "ascii-cfg" (PID 20023) hasn't caught signal 11 (core will be saved). 2021 Dec 11 05:54:21.252051 LEAF47 %$ VDC-1 %$ %SYSMGR-2-SERVICE_CRASHED: Service "ascii-cfg" (PID 26524) hasn't caught signal 11 (core will be saved). 2021 Dec 11 06:04:31.746874 LEAF47 %$ VDC-1 %$ %SYSMGR-SLOT1-2-SERVICE_CRASHED: Service "tahusd" (PID 23621) hasn't caught signal 11 (core will be saved). 2021 Dec 11 06:05:18.666840 LEAF47 %$ VDC-1 %$ %SYSMGR-SLOT1-2-LAST_CORE_BASIC_TRACE: core_client_main: PID 29910 with message filename = 0x102_tahusd_log.23621.tar.gz * We also see the following syslogs: high count of PFC frames reported %ACLQOS-SLOT1-2-ACLQOS_UNEXPECTED_PFC_FRAMES: Ethernet1/31 received 566935683072 unexpected PFC frames for COS 0

Conditions

using NXAPI-DME-REST to create or delete checkpoints on Nexus 9000 with EVPN VXLAN configuration

Workaround

pro-active: use NXAPI-CLI to create/delete/list checkpoints. corrective action: * if the switch has become completely unresponsive and did not reload by itself, a manual power cycling will restore operation

Further Problem Description