Symptom

We can see previous the crash the following logs: SWITCH_1_RP_0 smand[25503]: %PLATFORM-3-ELEMENT_CRITICAL: 1/RP/0: Used Memory value 99% exceeds critical level 95% SWITCH_1_RP_0 smand[25503]: %PLATFORM-3-ELEMENT_CRITICAL: 1/RP/0: Used Memory value 99% exceeds critical level 95% SWITCH_1_RP_0 smand[25503]: %PLATFORM-3-ELEMENT_CRITICAL: 1/RP/0: Used Memory value 98% exceeds critical level 95% Monitoring the active switch memory (in this example is 2) there is an increase in nginx : From Jan 08th to Jan 14th: SWITCH#show platform software process memory switch 2 r0 all sorted Pid VIRT RSS PSS Heap Shared Private Name ------------------------------------------------------------------------------------ 15697 234184 95380 91700 1260 5360 90020 nginx 15697 441816 302996 299316 1260 5360 297636 nginx 15697 458264 319436 315756 1260 5360 314076 nginx 15697 504108 365328 361648 1260 5360 359968 nginx 15697 532268 393524 389844 1260 5360 388164 nginx <<<<<<<<<<<+298MB

Conditions

Two condition: 1. See a meaningful increase in one callsite in section diff_call: SWITCH#show platform soft memory nginx switch active R0 alloc callsite brief The current tracekey is : 1#2e659cba8f5621a05fd7f543de428cdd callsite thread diff_byte diff_call ---------------------------------------------------------- 4170090497 15697 1382203648 84364 <<<<<<<<<<<<monitor this to see increase in this callsite 809741313 15697 57384 1 809698304 15697 57360 1 809692161 15697 16960 1 809741314 15697 11808 5 940158978 15697 1490 15 2. "ip http secure-server" needs to be configured in the device.

Workaround

Disable "ip http secure-server" in the configuration will decrease the memory utilization and no memory leak will be seen: 2021.03.23 09:40:38 HTTPS server was enabled again "ip http secure-server": Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 3-RP0 Healthy 3976844 2514580 (63%) 1462264 (37%) 3532944 (89%) <<<<<<<<<<<<< After two weeks "ip http secure-server" keep enabled we can see an increase from 63% to 86%: 2021.04.07 14:44:56 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 3-RP0 Healthy 3976844 3427932 (86%) 548912 (14%) 4448136 (112%) <<<<<<<<<<<< The same day, "ip http secure-server" was disabled and suddenly it decreases to 52%: 2021.04.07 14:48:54 Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) 3-RP0 Healthy 3976844 2074084 (52%) 1902760 (48%) 3059812 (77%) <<<<<<<<<<<<

Further Problem Description