Symptom
IOSXE router experiences rapid memory exhaustion and then crashes. Prior to the crash, memory utilization in cpp_cp_svr process rapidly increases in the module OSLib Mutex Objects:
Router# show platform software memory qfp-control-process qfp active
Module: OSLib Mutex Objects
allocated: 1949091456, requested: 1349371008, overhead: 599720448 << high overhead
Allocations: 37482528, failed: 0, frees: 0 << large number of allocations, 0 frees
Conditions
1. ACL rules configured with logging enabled:
deny log
permit log
2. ACL logging threshold not set or set to a low value (way below 1000):
ip access-list log-update threshold
This is to set the number of matches in ACL before a log message is sent to control plane.
3. Multiple ACL logs from same source IP seen in the log files (fman_fp btrace or bootup syslog)
Workaround
1. Reduce ACL logging on generic ACL rules like:
deny ip any any log
2. If logging has to be enabled add specific ACL rules:
deny ip any host P.Q.R.S log
3. If there are many ACL logs seen from same source IP (check fman_fp btrace decode or bootup log), add a specific rule without 'log' for that IP:
deny ip host A.B.C.D
4. The following three CLIs to be used and tuned to the system under consideration
ip access-list log-update threshold // 10 as an example
logging rate-limit // 100 as an example
ip access-list logging interval // 10 as an example
For details please refer -> https://sec.cloudapps.cisco.com/security/center/resources/access_control_list_logging.html
