OPERATIONAL DEFECT DATABASE
...
...
IOSd memory leak that manifests under *MallocLite*, which is a memory manager used to allocate small, fixed-size chunks of memory. ------------------ show processes memory sorted ------------------ Processor Pool Total: 1739968648 Used: 1393851988 Free: 346116660 <<<----------------- 80% Used in IOSd PID TTY Allocated Freed Holding Getbufs Retbufs Process 0 0 0 0 651338732 0 0 *MallocLite* <<<---------- Holding over 620 MB of memory 0 0 490802740 218876168 234648464 0 0 *Init* 0 0 172214451948 171069112432 159142812 16651080 0 *Dead* 684 0 1504620452848 1627043719440 119916988 45928 0 CCSIP_SPI_CONTRO 686 0 243292702160 104896168104 55251140 0 0 CCSIP_UDP_SOCKET 429 0 3441041244 3809698040 32602648 0 0 AFW_application_ 1 0 1735097244 1717287042 13806436 0 0 Chunk Manager TAC can check if a single PC in *MallocLite* is allocating a significant amount of memory (using the "show memory lite-chunks totals" command), and if so, which code that allocator is part of. This defect tracks a large, and growing, amount of memory allocated by the Smart Agent functions. On the problematic router, we have seen memory utilization increase from roughly 40% used to roughly 80% over the course of 10 weeks. If you believe you are running into this behavior, please collect the following outputs and open a TAC case: show clock show version show processes memory sorted show memory lite-chunks totals
A router or switch using the Smart Licensing Using Policy (SLP) model of licensing. This memory leak is seen when there's a buildup of unacknowledged reports on the device.
Reloading during a planned maintenance window will free the memory being held currently, essentially resetting the memory already leaked. However, this is only a temporary mitigation as it does not correct the source of the leak. Customers could also try switching to the offline method of SLP (rather than using OnPrem) to help avoid the generation of RUM reports periodically. For details, please refer to the "Workflow for Topology: No Connectivity to CSSM and No CSLU" section of the "Smart Licensing Using Policy for Cisco Enterprise Routing Platforms" guide: https://www.cisco.com/c/en/us/td/docs/routers/sl_using_policy/b-sl-using-policy/how_to_configure_workflows.html#Cisco_Concept.dita_7057e18c-3c69-4d91-841b-0b5beb7a2d88 The software fix is available starting in 17.3.7 and 17.6.5.
The accumulation of RUM reports can lead to high CPU and memory utilization. The fix for high CPU utilization was addressed in CSCwa85199: CSCwa85199 - Unacknowledged Reports can cause High CPU Utilization due to Smart Agent The fix for high memory utilization under MallocLite was addressed in CSCwa85525: CSCwa85525 - Memory leak in *MallocLite* due to growing Smart Agent Memory Utilization Starting in IOS-XE 17.8, the way SLP handles RUM reports has been updated, and this issue does not affect those newer releases. Please ensure that your licensing workflow is operating as expected. Devices should be able to sync with OnPrem, OnPrem should be able to sync with CSSM, and the acknowledgment (ACK) from CSSM should make its way back to OnPrem and ultimately onto the router.
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
