Symptom

SIP40 card gets reloaded unexpected after ran out of memory. below are syslog messages related to the issue. Oct 23 22:43:35.008: %OOM-3-NO_MEMORY_AVAIL: SIP0: oom: The system is very low on available memory. Operations will begin to fail. Oct 23 22:44:02.061: %IOSXE-5-PLATFORM: F0: cpp_cp: QFP:0.0 Thread:111 TS:00007532522254592104 %PUNT_INJECT-5-DROP_PUNT_CAUSE: punt cause policer drop packet casue 60 Oct 23 22:44:04.307: %IOSXE-4-PLATFORM: SIP0: kernel: systemd-journal invoked oom-killer: gfp_mask=0x800d0, order=0, oom_score_adj=0 Oct 23 22:44:04.332: %IOSXE-3-PLATFORM: SIP0: kernel: Out of memory: Kill process 4088 (sleep) score 998 or sacrifice child Oct 23 22:44:04.332: %IOSXE-3-PLATFORM: SIP0: kernel: Killed process 4088 (sleep) total-vm:2260kB, anon-rss:68kB, file-rss:492kB Oct 23 22:44:04.298: %OOM-3-NO_MEMORY_AVAIL: SIP0: oom: The system is very low on available memory. Operations will begin to fail. Oct 23 22:44:04.788: %IOSXE-4-PLATFORM: SIP0: kernel: cmcc invoked oom-killer: gfp_mask=0x80d0, order=0, oom_score_adj=133 Oct 23 22:44:04.800: %IOSXE-3-PLATFORM: SIP0: kernel: Out of memory: Kill process 4112 (sleep) score 998 or after rebooted, confirm cmcc is leaking #show platform software process list 0 | i cmcc cmcc 3740 3131 3740 S 20 70164480 #show platform software process list 0 cmcc 3744 3134 3744 S 20 70385664 <<<<< increased. 3 wks later

Conditions

When there is nfs link break issue we come across this kind of problem. The SIP40 card complains about memory consumption more than a certain threshold.

Workaround

Once the SIP40 card gets reloaded due to unexpected out of memory issue, its nfs link will get auto-corrected issue will get fixed.

Further Problem Description