Symptoms

Impact:HA not in sync.Unable to do Logical switch activity.Environment:Dell EMC Hardware: Connectrix ED-8510-4BDell EMC Hardware: Connectrix ED-8510-8BDell EMC Hardware: Connectrix ED-DCX6-4BDell EMC Hardware: Connectrix ED-DCX6-8BSoftware: Fabric OS v8.2.1c Issue:Maps daemon (MD) terminates causing CPs to go out of sync.Example of the termination: [KSWD-1002], 13255, SLOT 2 | FFDC | CHASSIS, WARNING, switchname, Detected termination of process mdd:2403 [FSSM-1003], 13256, SLOT 1 | CHASSIS, WARNING, switchname, HA State out of sync. [FSS-1008], 342411/13259, SLOT 2 | FID 128, CRITICAL, switchname, FSS Error: fssd_register failure fcsw:0:0:md on switch 0 … [HAM-1007], 13268, SLOT 2 | FFDC | CHASSIS, CRITICAL, switchname, Need to reboot the system for recovery, reason: Software Bootup Failure:LS config timed out. [HAM-1009], 13269, SLOT 2 | FFDC | CHASSIS, CRITICAL, switchname, Need to MANUALLY REBOOT the system for recovery - auto-reboot is disabled. Message when unable to do logical switch activity: The system is not ready for Logical Switch management operations. Example of HA out of sync: /fabos/cliexec/hadump: --------------------------------------- TIME_STAMP: Apr 27 20:28:59.532174 --------------------------------------- Local CP (Slot 2, CP1): Active, Warm Recovered Remote CP (Slot 1, CP0): Standby, Healthy HA enabled, Heartbeat Up, HA State not in sync

Cause

During normal operation, a rare race condition led to a process deadlock.Watchdog kills MDD daemon after MAPS daemon (MD) receives an IPC request from weblinker to enable "dflt_base_policy" policy.The worker thread spawned to enable the given MAPS policy holds the notification manager mutex lock and attempts to configure the relay host for send mail and it gets stuck.Any other worker thread waiting for the same lock times out and fails to refresh watchdog resulting in process termination.

Resolution

Resolution:The active CP must be recovered and rebooted in a maintenance window. The COLD reboot causes an outage on the switch.The fix is ported into Fabric OS 8.2.1d under DEFECT FOS-817272.

Support Cases