Symptom
Nexus 3500 running the version 9.3.8 unexpectedly rebooted due to a kernel panic
Last reset at 858020 usecs after x
Reason: Kernel Panic
System version: 9.3(8)
Service:
Conditions
The kernel panic was produced due to a high memory utilization.
nxos_panic: Kernel panic - not syncing: Out of memory: system-wide panic_on_oom is enabled
Kernel panic shows the process ipfib with high memory.
[927830.569782] [ pid ] uid tgid total_vm rss nr_ptes nr_pmds swapents oom_score_adj name
[927830.570317] [19252] 0 19252 323998 223834 489 3 0 0 ipfib
[927830.570293] [18810] 0 18810 128112 41976 154 3 0 0 mtc_cr_usd
[927830.570296] [18811] 0 18811 84653 40430 93 2 0 0 neutron_usd_uc
[927830.570007] [16275] 0 16275 120033 28167 90 3 0 0 urib
[927830.569898] [16219] 0 16219 91216 21046 72 3 0 0 clis
Checking the process ipfib in the line card
Private Mem stats for UUID : IPFIB Manager(378) Max types: 203
--------------------------------------------------------------------------------
TYPE NAME ALLOCS BYTES
CURR MAX CURR MAX
20 L2MCAST_MEM_l2mcast_di_t 39 50 3588 4600
22 L2MCAST_MEM_l2mcast_di_next_hop_t 38 50 2736 3600
23 L2MCAST_MEM_l2mcast_di_next_hop_port_ar 37 50 636 1080
24 L2MCAST_MEM_l2mcast_port_array_t 8994989 8994989 143647604 143647604 >>>> increasing
Workaround
Memleak introduced in the version 9.3.8.
To avoid this issue, 9.3.7 version can be used.
Further Problem Description
With the latest design change on MTC in 9.3.8, there is a memory leak in ipfib process by L2MCAST_MEM_l2mcast_port_array_t.
As part of this DDTS, the memleak has been resolved at all required places.
