Symptom

Mem leak observed for RPM Process, eventually crashing later 2022 Nov 24 03:36:16.734301 [hostname] rpm[28976]: Warning! Malloc Failure in ../routing-sw/libs/timers/active_timer.c[928] for size 8196 2022 Nov 24 03:36:16.125750 [hostname] %RPM-3-ATIMERS_ERROR: malloc failed in heap_create 2022 Nov 24 03:36:16.126152 [hostname] %RPM-3-ERROR: -Traceback: librsw.so=0xf6b82000 librsw.so+0xe8fd4... 2022 Nov 24 03:36:16.126176 [hostname] %RPM-3-INFRA_SYSERR: rpm [28976] ppf_srv_ses_verify failed with error - Memory allocation (0x41170042) - in rpm_ppf_verify() 2022 Nov 24 03:36:16.127493 [hostname] %RPM-4-SYSLOG_SL_MSG_WARNING: RPM-3-INFRA_SYSERR: message repeated 1 times in last 122220518 sec 2022 Nov 24 03:40:12.084464 [hostname] %SYSMGR-2-SERVICE_CRASHED: Service "rpm" (PID 28976) hasn't caught signal 11 (core will be saved).

Conditions

Due to RPM Heap Memory exhaustion, RPM process is restarted. As a after effect, PPF Sessions were not getting opened and below syslogs are seen, 2022 Nov 20 02:52:51.578893 [hostname] %RPM-3-INFRA_SYSERR: rpm [31411] ppf_srv_ses_open() failed with error - Invalid operation (0x4117000c) - in rpm_ppf_ses_open()

Workaround

Since this is a mem leak issue, Safe workaround is to upgrade to 7.0(3)I7(3) or later CCO Images. Also memory leaks happens only for PBR Related configs, it can be removed to stop memory leaks.

Further Problem Description