Symptom

Elevated CPU usage (80%-90%+) is observed on the Secure Firewall Threat Defense (FTD) system cores, depending on the eventing rate. The Secure Firewall Management Center (FMC) displays a warning or critical alert "CPU Usage (per core) ? CPUXX (system) usage is YY", where CPUXX is the system core, YY is the CPU utilization. On FTD, the eventing rate and CPU utilization can be checked in /ngfw/var/log/EventHandlerStats. files. In this example, the event destinations are 2 FMCs in high availability (HA), SecureX cloud via SSEConnector, and the syslog servers (logging to syslog server is configured in an access control rule): admin@firepower:/ngfw/Volume/home/admin$ less /ngfw/var/log/EventHandlerStats.2024-01-28 {"Time": "2024-02-03T23:58:32Z", "TotalEvents": 11455672, "PerSec": 38156, "UserCPUSec": 284.238, "SysCPUSec": 74.676, "%CPU": 119.5, "MemoryKB": 73704} <---------- Total event rate and CPU utilization by eventing. {"Time": "2024-02-03T23:58:32Z", "Consumer": "SSEConnector", "Events": 2943921, "PerSec": 9806, "CPUSec": 222.335, "%CPU": 74.1} <---------- Eventing to SecureX cloud using SSEConnector {"Time": "2024-02-03T23:58:32Z", "Consumer": "UECPeerLow_04352360-507b-11ee-9d3d-54bc5fd42fd9", "Events": 3034221, "PerSec": 10107, "AggregationEvents": 20, "CPUSec": 22.038, "%CPU": 7.3, "OutputWaitSec": 150.505, "ProcessingWaitSec": 147.667} <---------- Low priority eventing to FMC1 {"Time": "2024-02-03T23:58:32Z", "Consumer": "Syslog", "Events": 2245205, "PerSec": 7478, "CPUSec": 84.439, "%CPU": 28.1} <---------- Eventing to syslog servers {"Time": "2024-02-03T23:58:32Z", "Consumer": "UECPeerLow_13aef37a-5080-11ee-ba58-9a669468247e", "Events": 3035079, "PerSec": 10110, "AggregationEvents": 20, "CPUSec": 22.097, "%CPU": 7.4, "OutputWaitSec": 153.079, "ProcessingWaitSec": 150.842} <---------- Low priority eventing to FMC2 {"Time": "2024-02-03T23:58:32Z", "Consumer": "UECPeerHigh_13aef37a-5080-11ee-ba58-9a669468247e", "Events": 98760, "PerSec": 328, "CPUSec": 3.575, "%CPU": 1.2} <---------- High priority eventing to FMC1 {"Time": "2024-02-03T23:58:32Z", "Consumer": "UECPeerHigh_04352360-507b-11ee-9d3d-54bc5fd42fd9", "Events": 98487, "PerSec": 327, "CPUSec": 3.520, "%CPU": 1.2} <---------- High priority eventing to FMC2

Conditions

First seen on FTD with the event rate of 20000-40000 or higher event per sec, and event logging to FMC in HA, SecureX cloud, and unified event logging to one or more syslog servers. Similar symptoms may be observed in different configurations depending on the eventing rate and configuration.

Workaround

To reduce CPU utilization, perform one or more of the following steps: 1. Disable logging of specific events (for example, connection events) to SecureX cloud. 2. Disable logging to SecureX cloud. 3. Reduce event logging (for example, disable logging on specific access control rules with the highest hit count).

Further Problem Description