Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
N9k running versions 9.3(1) to 9.3(4) may experience a memory leak on the aclqos service. There are two ways you might experience this. Memory utilization increasing on the switch. Upon inspecting the output of "show system internal aclqos mem-stats detail | no" you may find the utilization on "libindex_mgmt.so" Private Mem stats for UUID : Non mtrack users(0) Max types: 218 -------------------------------------------------------------------------------- TYPE NAME ALLOCS BYTES CURR MAX CURR MAX 0 [0x100a8000]aclqos 0 2 0 29 ... 101 [0xf5b40000]libindex_mgmt.so 302 302 74092 74092 ... Eventually, the aclqos process may crash reporting the following in the log: 2021 Mar 27 10:41:05.332 %SYSMGR-2-SERVICE_CRASHED: Service "aclqos" (PID 5757) hasn't caught signal 11 (core will be saved). 2021 Mar 27 10:41:05.389 %SYSMGR-2-LAST_CORE_BASIC_TRACE: : PID 7363 with message aclqos(non-sysmgr) crashed, core will be saved Contact cisco TAC and give core file so that they can verify the same leak on it.
The issue is seen when TCP-NAT is configured on the switch. With TCP-NAT, when new NAT translation sessions are created / deleted there is a possibility of memory leak. If such NAT sessions are created / deleted for long enough, enough memory can leak causing aclqos process to crash the issue also can happen on N3548 device without TCP-NAT config, with the link flapping, it will trigger "aclqos" memory leak happened as below: N3548# show clock ; show processes memory | sec aclqos 23:45:59.559 HKT Wed Mar 30 2022 Time source is NTP 31863 168132608 0 494366720 ffe818e0/ffe808d0 aclqos N3548# show clock ; show processes memory | sec aclqos 00:18:12.130 HKT Thu Mar 31 2022 Time source is NTP 31863 171106304 0 497340416 ffe818e0/ffe808d0 aclqos
Monitor the outputs of "show process memory" and "show system internal aclqos mem-stats detail | no". If you find the utilization on "libindex_mgmt.so" increasing, reload the switch before aclqos reaches it's process limit (according to show process memory). Permanent fix is to upgrade to a fixed release.