Symptom
++ We see Memory Leak happening in the "fed main event" process.
The RSS/Total counters keep increasing over time in the below output:
Switch#show processes memory platform sorted
System memory: 7711588K total, 6886612K used, 824976K free,
Lowest: 824976K
Pid Text Data Stack Dynamic RSS Total Name
--------------------------------------------------------------------------------
16433 145 4006780 136 85064 4006780 7059676 fed main event <----
14490 144051 892456 136 296 892456 4632172 linux_iosd-imag
14162 270 50604 136 2072 50604 1646600 sif_mgr
14383 739 61548 136 1652 61548 1519636 platform_mgr
23529 6710 122256 136 2916 122256 1461768 fman_rp
++ The following errors can be seen in the FED tracelogs:
Switch# show logging process fed start last boot
2020/05/27 04:56:34.400 {fed_F0-0}{1}: [asic_fnf] [16453]: UUID: 0, ra: 0, TID: 0 (ERR): AFD-INFRA:FNF Failed to read flow table
2020/05/27 04:56:34.400 {fed_F0-0}{1}: [asic_nfl] [16453]: UUID: 0, ra: 0, TID: 0 (ERR): AFD-INFRA:NFL curr_fmask_entry is NULL! Mask-Id 2 Lut 1, asic 0
Conditions
Catalyst 9000 switch with the same flow monitor configured on a port-channel and regular interface.
The switch is running the IOSXE image that already has the fix for similar bug tracked under CSCvs50868.
Workaround
Remove flow monitors from all interfaces and reload the device. Once the device is back up, configure a separate flow monitors on the regular interfaces and on the port-channel member interfaces.
However, this still might not fix the memory leak, and further deep dive is needed.
