Symptom

C8500 Carrier Grade NAT reaching max host entries and failing to translate Reaching a NAT host limit after which point the new flows coming in from that same host cannot be translated. This results in a gatekeeper cache entry. Possible because the DNS traffic is commonly creating new flows on different ports, it is common for this traffic to come in when the host limit is exceeded and thus create a gatekeeper cache entry. Now the gatekeeper cache only cares about source and destination IP. If there is never a 10 second gap during which no DNS traffic comes in for that src/dest, the gatekeeper cache entry will remain. All new DNS requests to be dropped.

Conditions

As detailed by TAC: Log messages indicating max host entries being exceeded along with the gatekeeper cache entries for sources which should always be translated. From troubleshooting we can see that traffic is forwarded without translation due to GATEKEEPER_BLOCKED as shown in the NAT FIA: Example: >Packet trace for a failing ping from 172.26.39.7 (ONT nat ip) to 189.194.224.51 (dns server) > We can see that the c8500 is not translating this flow. N_C8500_2_AGS#sh plat packet-trace pac 0 Packet: 0 CBUG ID: 57370 Summary Input : Port-channel47.11 Output : Port-channel47.34 State : FWD Timestamp Start : 4762214244691663 ns (10/19/2023 10:54:32.65092 UTC) Stop : 4762214245039081 ns (10/19/2023 10:54:32.65439 UTC) Path Trace Feature: IPV4(Input) Input : Port-channel47.11 Output : Source : 172.26.39.7 Destination : 189.194.224.51 Protocol : 1 (ICMP) Feature: NAT Direction : IN to OUT From : NON-SDWAN Action : FWD without translation FWD-POINT : GATEKEEPER_BLOCKED VRFID : 0 table-id : 0 Protocol : ICMP Src Addr : 172.26.39.7 Dest Addr : 189.194.224.51 src port : 3 dst port : 3 Feature: IPV4_NAT_OUTPUT_FIA Entry : Output - 0x800188c8 Input : Port-channel47.11 Output : Port-channel47.34 Lapsed time : 29776 ns Feature: IPV4_OUTPUT_THREAT_DEFENSE Entry : Output - 0x800188d0 Input : Port-channel47.11 Output : Port-channel47.34 Lapsed time : 352 ns Feature: IPV4_VFR_REFRAG Entry : Output - 0x800185c8 Input : Port-channel47.11 Output : Port-channel47.34 Lapsed time : 208 ns Feature: DEBUG_COND_APPLICATION_OUT_CLR_TXT Entry : Output - 0x800181f4 Input : Port-channel47.11 Output : Port-channel47.34 Lapsed time : 272 ns Feature: Etherchannel Port-channel : 47 Algorithm : Flow based : Source and Destination IPV4 SRC IP : 172.26.39.7 DST IP : 189.194.224.51 SRC MAC : 5049.21be.5a77 DST MAC : e8d3.22d3.46ee Bundle id : 47 Bucket id : 3 Egress Link : TenGigabitEthernet0/0/4

Workaround

Further Problem Description