Symptom

Customer’s all ISR1100-6G cEdge routers have the process ?cpp_cp_svr_fp_0? as the top memory consumer. And ?show processes memory platform accounting? shows ?max_diff_bytes?/?max_diff_calls? for this process increasing on a daily basis and leading to crash. Feb 20 18:16:16.519: %PLATFORM-4-ELEMENT_WARNING: R0/0: smand: RP/0: Used Memory value 93% exceeds warning level 88%. Top memory allocators are: Process: cpp_cp_svr_fp_0. Tracekey: 1#6eddd142fcebb6a120bc73dad5a66f1e Callsite ID: 3169153026 (diff_call: 1422699). Process: keyman_rp_0. Tracekey: 1#1afb75f86df75f6d98969953a96fb3ab Callsite ID: 2766961665 (diff_call: 103991). Process: cpp_ha_top_leve_fp_0. Tracekey: 1#cd96bc1864f4deb02294e888126c177c Callsite ID: 3169153026 (diff_call: 99604). cedge#show platform software status control-processor brief Memory (kB) Slot Status Total Used (Pct) Free (Pct) Committed (Pct) RP0 Critical 3878084 3622972 (93%) 255112 ( 7%) 4926236 (127%) - Committed memory is showing 127% while used memory is 93% cedge-s#show platform software process memory r0 all sorted Pid RSS PSS Heap Shared Private Name -------------------------------------------------------------------------- 18196 891716 743728 160 221244 670472 cpp_cp_svr cedge#show processes memory platform accounting Daily Stats process callsite_ID(bytes) max_diff_bytes callsite_ID(calls) max_diff_calls tracekey timestamp(UTC) ------------------------------------------------------------------------------------------------------------------------------------------------------------ cpp_cp_svr_fp_0 3169153026 470549884 3169153026 1424061 1#6eddd142fcebb6a120bc73dad5a66f1e 2023-02-21 07:37 linux_iosd-imag_rp_0 2428000260 17653962 1956637699 4677 1#1a46a3642b8691e0eaeee02b9a6b16f3 2023-02-19 01:46 cpp_ha_top_leve_fp_0 3169153026 9622248 3169153026 100228 1#cd96bc1864f4deb02294e888126c177c 2023-02-21 06:31 keyman_rp_0 2766961665 6712704 2766961665 104886 1#1afb75f86df75f6d98969953a96fb3ab 2023-02-21 06:33 March 14th for xxxx device:- Daily Stats process callsite_ID(bytes) max_diff_bytes callsite_ID(calls) max_diff_calls tracekey timestamp(UTC) ------------------------------------------------------------------------------------------------------------------------------------------------------------ cpp_cp_svr_fp_0 3169153026 120509788 3169153026 363891 1#6eddd142fcebb6a120bc73dad5a66f1e 2023-03-14 00:13 March 16th process callsite_ID(bytes) max_diff_bytes callsite_ID(calls) max_diff_calls tracekey timestamp(UTC) ------------------------------------------------------------------------------------------------------------------------------------------------------------ cpp_cp_svr_fp_0 3169153026 128303789 3169153026 387528 1#6eddd142fcebb6a120bc73dad5a66f1e 2023-03-15 00:16 March 17th process callsite_ID(bytes) max_diff_bytes callsite_ID(calls) max_diff_calls tracekey timestamp(UTC) ------------------------------------------------------------------------------------------------------------------------------------------------------------ cpp_cp_svr_fp_0 3169153026 136312019 3169153026 411783 1#6eddd142fcebb6a120bc73dad5a66f1e 2023-03-16 01:13

Conditions

Only when running tech-support or NBAR debug commands.

Workaround

Each run generates a small memory leak, so refrain from executing ?tech-support? or NBAR debug commands frequently.

Further Problem Description