Symptom

A Firepower device might stop offloading all existing and new flows. During the time of the issue the symptoms are: Symptom 1 - The application CPU (e.g. FTD) increases since the flows are not any more offloaded. The CPU increase depends on the amount of the traffic that was offloaded: # show cpu CPU utilization for 5 seconds = 54%; 1 minute: 53%; 5 minutes: 52% Symptom 2 Flows are not offloaded any more (static or dynamic flow offload). This can be confirmed in multiple ways. Way 1 firepower# show flow-offload statistics Packet stats of port : 0 Tx Packet count : 151306479486 <-- this counter either is not increasing or is increasing very slowly Rx Packet count : 151306479486 <-- this counter either is not increasing or is increasing very slowly Dropped Packet count : 0 VNIC transmitted packet : 151306479486 VNIC transmitted bytes : 140891960082156 VNIC Dropped packets : 0 VNIC erroneous received : 0 VNIC CRC errors : 0 VNIC transmit failed : 0 VNIC multicast received : 0 Way 2 The Bytes of an offloaded flow are not increasing any more: firepower# show flow-offload flow address 192.0.2.1 port 12345 address 192.0.2.2 port 67890 TCP intfc 1021 src 192.0.2.1:12345 dest 192.0.2.2:67890, dynamic, ft index 1072111 timestamp 580053164, packets 37990627279, bytes 54591204082787 <-- ... TCP intfc 1021 src 192.0.2.2:67890 dest 192.0.2.1:12345, dynamic, ft index 2847222 timestamp 580053166, packets 1893063459, bytes 142061644406 <--- After a few seconds the values are still the same: firepower# show flow-offload flow address 192.0.2.1 port 12345 address 192.0.2.2 port 67890 TCP intfc 1021 src 192.0.2.1:12345 dest 192.0.2.2:67890, dynamic, ft index 1072111 timestamp 580053164, packets 37990627279, bytes 54591204082787 <-- ... TCP intfc 1021 src 192.0.2.2:67890 dest 192.0.2.1:12345, dynamic, ft index 2847222 timestamp 580053166, packets 1893063459, bytes 142061644406 <--- Symptom 3 - There is a mismatch between the 'compare' column of 'ft-show_config 0' output and the 'flow table refresh count' counter of 'show flow-offload info detail' output. Normally, the 'compare' column value is the modulus 128 of the 'flow table refresh count' ('flow table refresh count' mod 128), e.g. firepower# show flow-offload info detail | i refresh flow table refresh count : 634 vs Firepower# connect adapter 1/1/1 adapter 1/1/1 # connect adapter 1/1/1 (top):1# attach-mcp adapter 1/1/1 (mcp):17# ft-show_config 0 ... ------------------------------------------------------------------------------------------- ft_id ebase en auto netflow bucket bbase hit miss coll compare insert ------------------------------------------------------------------------------------------- 0 7b017000 262144 1 1 1 7ae17000 0 0 0 0 0 1 7c017000 65536 1 1 1 7af17000 0 0 0 0 0 2 7c417000 65536 1 1 1 7af57000 0 0 0 0 0 3 7c817000 65536 1 1 1 7af97000 0 0 0 0 0 4 7cc17000 65536 1 1 1 7afd7000 0 0 0 0 0 5 55e17000 4194304 0 1 0 0 1440 0 0 122 0 <--- 6 55e17000 4194304 0 1 0 0 1440 0 0 122 0 <--- 8 55e17000 4194304 0 1 0 0 1440 0 0 122 0 <--- 9 55e17000 4194304 0 1 0 0 1440 0 0 122 0 <--- In this case, 634 mod 128 = 122 Symptom 4 When the issue is triggered there is an increase in the 'flow table refresh count' value. Note that the increase does not always trigger the issue, but whenever the issue is seen there is an increase just before it. # show flow-offload info detail Current running state : Enabled User configured state : Enabled Dynamic flow offload : Enabled Offload App : Running flow table refresh count : 61 <---- Offload allocated cores : S0[ 2] Offload Nic : 9 Max PKT burst : 32 Port-0 details : RX queue number : 149 FQ queue number : 1440 Keep alive counter : 76878 # show flow-offload info detail Current running state : Enabled User configured state : Enabled Dynamic flow offload : Enabled Offload App : Running flow table refresh count : 105 <---- Offload allocated cores : S0[ 2] Offload Nic : 9 Max PKT burst : 32 Port-0 details : RX queue number : 149 FQ queue number : 1440 Keep alive counter : 76878 This counter increases when one of the following occurs: 1. Route update (change in routing path) 2. ECMP path selection change 3. MAC address change 4. Interface shut/no shut 5. Clear interface 6. HA (Failover) event 7. Interface flap

Conditions

Workaround

Reload the logical device

Further Problem Description