Symptom

FMC becomes unresponsive (cannot SSH into or use the GUI) with error logs shown below. The failure may occur after 2-4 weeks. --------------------------------------------------------------------------------------------------------------------------------------------- Sep 5 15:54:21 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [ERROR] Unable to create command handler thread: Resource temporarily unavailable Sep 5 15:54:22 dc10-fmc-107 SF-IMS[14822]: [19435] sfmgr:rexecchild [ERROR] Ran out of memory to fork SFMGR Sep 5 15:57:27 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [ERROR] Unable to create command handler thread: Resource temporarily unavailable Sep 5 16:00:27 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [WARN] Remote task (fd=8, uuid=945a487a-de74-11ea-af96-e2acd58b909a,task_id_remote=33 : task_id_local=32) (sp=0x7fa060000960)exited and the peer send another one!! Sep 5 16:00:27 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [ERROR] Unable to create command handler thread: Resource temporarily unavailable Sep 5 16:03:28 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [WARN] Remote task (fd=8, uuid=945a487a-de74-11ea-af96-e2acd58b909a,task_id_remote=34 : task_id_local=33) (sp=0x7fa060005290)exited and the peer send another one!! Sep 5 16:03:28 dc10-fmc-107 SF-IMS[14822]: [19406] sfmgr:sfmanager [ERROR] Unable to create command handler thread: Resource temporarily unavailable Sep 5 15:56:03 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:08 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:13 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:18 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:23 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:28 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:33 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:38 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:43 dc10-fmc-107 init: cannot fork, retry.. Sep 5 15:56:48 dc10-fmc-107 init: cannot fork, retry.. ep 5 15:55:32 dc10-fmc-107 SF-IMS[13078]: [15603] CloudAgent:BeakerUtil [INFO] Types(1): Sep 5 15:53:38 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [ERROR] Failed to fork and exec monetdb_monitor: Resource temporarily unavailable Sep 5 15:53:38 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [WARN] Failed to fork and exec monetdb_monitor Sep 5 15:53:49 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [ERROR] Failed to fork and exec Syncd: Resource temporarily unavailable Sep 5 15:53:49 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [WARN] Failed to fork and exec Syncd Sep 5 15:54:12 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [ERROR] Failed to fork and exec TelemetryApp: Resource temporarily unavailable Sep 5 15:54:12 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [WARN] Failed to fork and exec TelemetryApp Sep 5 15:56:40 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [ERROR] Failed to fork and exec store_history: Resource temporarily unavailable Sep 5 15:58:36 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [ERROR] Failed to fork and exec CSDApp: Resource temporarily unavailable Sep 5 15:58:36 dc10-fmc-107 SF-IMS[12833]: [12833] pm:process [WARN] Failed to fork and exec CSDApp

Conditions

N/A

Workaround

Rebooting the FMC will recover from this lockup failure. And workaround is to disable the Cisco Support Diagnostics from the FMC Smart Licensing page.

Further Problem Description