Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
The NSX Edge cli provides detailed stats for Gateway interfaces. This KB provides details on how to interpret these stats and recommended actions that can be taken.Example of Edge cli Edge1> get logical-router interface 8638183a-7126-4401-b8cb-175bfce4664e stats Wed Jan 24 2024 UTC 07:09:38.729 interface : 8638183a-7126-4401-b8cb-175bfce4664e ifuid : 298 VRF : 5dfb911e-7c35-4c8d-b42c-12323ade5fdb name : tier0-interface-90-90-90-2 IP/Mask : 90.90.90.2/24;2010::250:56ff:fea6:cb7c/64(A);2010::1/64(A);fe80::250:56ff:fea6:cb7c/64(A) MAC : 00:50:56:a6:cb:7c VLAN : 8 LS port : 7ea2eb90-b82b-4f74-9349-ac9b2244af9f urpf-mode : STRICT_MODE admin : up op_state : up MTU : 1500 statistics RX-Packets : 275749 RX-Bytes : 23938078 RX-Drops : 4996 Blocked : 0 DST-Unsupported: 4515 Firewall : 0 Malformed : 0 No-Receiver : 0 No-Route : 0 RPF-Check : 81 Protocol-Unsupported: 396 IPv6 : 4 Port-Unsupported: 0 TTL-Exceeded: 0 Kni : 0 Rate-Limit : 0 IPsec : 0 IPsec-NoSA : 0 IPsec-NoVTI : 0 TX-Packets : 203743 TX-Bytes : 17505975 TX-Drops : 0 Blocked : 0 Firewall : 0 Frag-Needed : 0 No-neighbor : 0 No-Memory : 0 No-Linked-Port: 0 Rate-Limit : 0 IPsec : 0 IPsec-NoSA : 0 IPsec-NoVTI : 0 IPsec-Policy-Error: 0 IPsec-Policy-Block: 0 IP Ressemble Fragments-OK: 0 Fragemnts-Error: 0 Fragments-Timeout: 0 IP Fragment Fragments-OK: 0 Fragments-Error: 0
RX Interface Stats Stats (Rx)DescriptionActionBlockedPacket dropped due to interface being blocked/admin-down due to SR of the edge node is in standby state Some packets dropped due to blocked state is expected during failover/fallback case. if this counter continue to increase, the following command should be used to check admin and internal_operation: nsx> get logical-router interface c61a8048-21bf-498a-88e2-4a4a145d9244 | json { "admin": "up", "internal_operation": "up", } DST-UnsupportedPacket dropped due to unsupported destination like mcast (but mcast is not enabled), loopback or reserved address Start capture command can be used to find out the unsupported destination address and where is the traffic coming from (source address) start capture interface c61a8048-21bf-498a-88e2-4a4a145d9244 FirewallPacket dropped due to firewallCheck if there is drop rule or state mismatch traffic Edge1> get firewall b7d57745-8604-4d3c-b93d-f63164c8752b ruleset type firewall rules Mon Feb 12 2024 UTC 03:43:32.837 Rule count: 1 Rule ID : 1005 Rule : inout protocol any stateless from any to any accept Edge1> get firewall b7d57745-8604-4d3c-b93d-f63164c8752b ruleset type firewall stats Mon Feb 12 2024 UTC 03:43:20.547 Rule count: 1 Rule ID : 1005 Input bytes : 253299828 Output bytes : 275434116 Input packets : 4189054 Output packets : 4607874 Evaluations : 8796932 Hits : 8796928 Active connections : 0 MalformedPacket dropped due to malformed fields, ip checksum or L4 checksum could be the reason of malformed trafficCheck the sender checksum offload configurationNo-ReceiverPacket dropped due to destination being a lrouter with no receiver i.e. proto not supported by it or no linked tunnel existsCheck configuration of lrouter port, if its GRE port, then it should be linked to GRE tunnel port. root@Edge1:~# edge-appctl -t /var/run/vmware/edge/dpd.ctl lrouter_port/show 2f406d88-2515-4eba-a84b-7d2a1f05195d | json_pp { "ifuuid" : "2f406d88-2515-4eba-a84b-7d2a1f05195d", ... "lrouter" : "feca5512-4bb4-4346-819c-fd6a9bcac289", ... "name" : "test-gre - 0", ... "peer" : "37c4f125-84e0-494e-abc5-a941ecf7be8c", >>>> This should be GRE tunnel port UUID ... "ptype" : "gre-port", } Edge1> get tunnel-port 37c4f125-84e0-494e-abc5-a941ecf7be8c Mon Feb 12 2024 UTC 03:29:41.339 Tunnel : 37c4f125-84e0-494e-abc5-a941ecf7be8c IFUID : 394 LOCAL : 40.40.40.1 REMOTE : 5.5.5.5 ENCAP : GRE GRETAP VRFID: 2 No-RoutePacket dropped due to routing failure or invalid egress port Check L3 forwarding table for dst IP nsx> get logical-router 1eef3979-1bfb-4b41-b936-cf3451158d6c forwarding Fri Jan 26 2024 PST 13:55:51.367 Logical Router UUID VRF LR-ID Name Type 1eef3979-1bfb-4b41-b936-cf3451158d6c 2 8 SR-T0-PIAS-IDMZ SERVICE_ROUTER_TIER0 IPv4 Forwarding Table IP Prefix Gateway IP Type UUID Gateway MAC 0.0.0.0/0 10.10.138.14 route 3c71fae6-1a78-4203-92ef-ff996ca7a23f 00:1c:73:00:00:95 10.10.138.10 59b73791-3234-422e-b83e-0bef7283c0cc 00:1c:73:00:00:95 RPF-CheckPacket dropped due to no reverse path to destination Check RPF configuration (urpf-mode) and L3 forwarding table for SRC IP Edge1> get logical-router interface 2f406d88-2515-4eba-a84b-7d2a1f05195d Mon Feb 12 2024 UTC 03:37:45.617 interface : 2f406d88-2515-4eba-a84b-7d2a1f05195d ifuid : 395 VRF : feca5512-4bb4-4346-819c-fd6a9bcac289 name : test-gre - 0 mode : lif IP/Mask : 172.16.10.1/24 Fwd-mode : IPV4_AND_IPV6 MAC : 02:50:56:01:00:00 LS port : 37c4f125-84e0-494e-abc5-a941ecf7be8c urpf-mode : STRICT_MODE >>>>>>>>>>>>> admin : up op_state : up MTU : 1476 arp_proxy : Edge1> get logical-router feca5512-4bb4-4346-819c-fd6a9bcac289 forwarding Mon Feb 12 2024 UTC 03:38:14.600 Logical Router UUID VRF LR-ID Name Type feca5512-4bb4-4346-819c-fd6a9bcac289 2 3 SR-tier0 SERVICE_ROUTER_TIER0 IPv4 Forwarding Table IP Prefix Gateway IP Type UUID Gateway MAC 1.1.1.0/25 100.64.1.1 route 61ca295f-14b7-5c89-a6e0-f4bee9a153be 1.1.2.0/24 100.64.1.1 route 61ca295f-14b7-5c89-a6e0-f4bee9a153be 1.1.3.0/24 route e6807a42-0100-4776-b47e-90fbfc5c81ba 1.1.3.1/32 route 73555c8d-033b-5bd7-ae7d-cdc315fdb072 Protocol-UnsupportedPacket dropped due to known protocol like ARP, ICMP, DHCP but cannot be decoded completelyNo action unless there is traffic connection issue, such as no arp, no icmp reply and etc.TTL-ExceededPacket dropped due to TTL exceedingThere is a L3 loop, check forwarding information of the complete setupKniPacket dropped due to lrouter port's companion KNI port failed to sendCheck cpu utilization of linux process such as L7 LB, this is due to linux side not able to dequeue the pkt fast enough. # top top - 03:18:57 up 8 days, 7:17, 1 user, load average: 1.71, 2.20, 2.32 Tasks: 238 total, 3 running, 235 sleeping, 0 stopped, 0 zombie %Cpu(s): 7.9 us, 13.6 sy, 0.0 ni, 77.9 id, 0.1 wa, 0.0 hi, 0.5 si, 0.0 st KiB Mem : 7842940 total, 136860 free, 5437616 used, 2268464 buff/cache KiB Swap: 0 total, 0 free, 0 used. 2334760 avail Mem PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ TGID COMMAND 2784801 nsx-sha 20 0 3212 1004 916 S 0.0 0.0 0:00.00 2784801 sleep 2784770 root 20 0 8056 4100 3264 R 1.9 0.1 0:00.39 2784770 top 2784735 root 20 0 5168 4244 3504 S 0.0 0.1 0:00.04 2784735 bash Rate-LimitPacket dropped due to ingress traffic rate higher than configured QOS (traffic contract) Check if ingress traffic to the lrouter is much higher than the QOS config. You can find the QOS configured using the below CLI, Edge1> get logical-router 558ba208-8ae9-4b74-a379-55d94e3577ed qos-config Mon Jan 29 2024 UTC 06:16:45.835 Logical Router UUID : 558ba208-8ae9-4b74-a379-55d94e3577ed Direction : Ingress Rate (Mbps) : 1 Burst (bytes) : 1 IPsecPacket dropped during IPsec input processing Aggregated Rx drop counter for IPsec related drop, further check IPsec tunnel stats for particular drop reason using following command "get ipsecvpn tunnel stats <IPsec tunnel/VTI UUID>" edge1> get ipsecvpn tunnel stats cec70165-6e00-4ae8-93ef-b0338614a5e3 Interface UID : 295 Interface UUID : cec70165-6e00-4ae8-93ef-b0338614a5e3 VTI UUID : cec70165-6e00-4ae8-93ef-b0338614a5e3 Stats Rx Pkts : 0 Tx Pkts : 0 Rx Bytes : 0 Tx Bytes : 0 Rx MSS Adjusted : 0 Tx MSS Adjusted : 0 Rx MSS Ignored : 0 Tx MSS Ignored : 0 Rx Drops : 0 Tx Drops : 0 Rx Drop Crypto Failure : 0 Tx Drop Crypto Failure : 0 Rx Drop Enqueue Failure : 0 Tx Drop Enqueue Failure : 0 Rx Drop State Mismatch : 0 Tx Drop State Mismatch : 0 Rx Drop Malformed : 0 Tx Drop Malformed : 0 Rx Drop Proto Not Supported : 0 Tx Drop Proto Not Supported : 0 Rx Drop Replay : 0 Tx Drop Seq Rollover : 0 Rx Drop Inner Malformed : 0 Tx Drop Fragmentation Needed : 0 Rx Drop Policy Nomatch : 0 Rekey Request Failure : 0 Rx Drop Auth Failure : 0 Rx Drop Zero Sequence Number : 0 v6 Stats Rx Pkts : 3236 Tx Pkts : 3184 Rx Bytes : 250276 Tx Bytes : 631712 Rx MSS Adjusted : 0 Tx MSS Adjusted : 0 Rx MSS Ignored : 0 Tx MSS Ignored : 0 Rx Drops : 0 Tx Drops : 0 Rx Drop Crypto Failure : 0 Tx Drop Crypto Failure : 0 Rx Drop Enqueue Failure : 0 Tx Drop Enqueue Failure : 0 Rx Drop State Mismatch : 0 Tx Drop State Mismatch : 0 Rx Drop Malformed : 0 Tx Drop Malformed : 0 Rx Drop Proto Not Supported : 0 Tx Drop Proto Not Supported : 0 Rx Drop Replay : 0 Tx Drop Seq Rollover : 0 Rx Drop Inner Malformed : 0 Tx Drop Fragmentation Needed : 0 Rx Drop Policy Nomatch : 0 Rekey Request Failure : 0 Rx Drop Auth Failure : 0 Rx Drop Zero Sequence Number : 0 ---------------------------------------------------------------------------------------------------- IPsec-NoSAPacket dropped due to missing IPsec inbound or ingress security association SA Check Outbound and Inbound SAs installed by running below commands "get ipsecvpn sad" or "get ipsecvpn sad summary" edge1> get ipsecvpn sad summary Outbound SAs (6): ---------------------------------------------------------------------------------------------------------------------------- VRF Rule ID Src IP Dest IP Src Subnet Dest Subnet SPI NAT Rem Life ---------------------------------------------------------------------------------------------------------------------------- 2 1408217139 5050::100 2424::101 0xc6a6b127 N 1642 sec 2 334475315 5050::100 2424::101 0xcbe1de7e N 1581 sec 2 536870913 1111::10 1111::20 192.168.1.0/24 192.168.5.0/24 0x2e733800 N 922 sec 5 536870927 192.168.51.100 20.20.20.101 192.168.2.0/24 192.168.7.0/24 0xce11f2c1 N 2279 sec 2 1610612738 1111::10 1111::20 fec0:1::/64 fec0:5::/64 0x2786ae00 N 880 sec 5 1610612752 192.168.51.100 20.20.20.101 fec0:2::/64 fec0:7::/64 0xc02800f4 N 2279 sec Inbound SAs (6): ---------------------------------------------------------------------------------------------------------------------------- VRF Rule ID Src IP Dest IP Src Subnet Dest Subnet SPI NAT Rem Life ---------------------------------------------------------------------------------------------------------------------------- 5 2684354575 20.20.20.101 192.168.51.100 192.168.7.0/24 192.168.2.0/24 0xbf806100 N 2279 sec 2 3555700787 2424::101 5050::100 0xc3061f00 N 1642 sec 5 3758096400 20.20.20.101 192.168.51.100 fec0:7::/64 fec0:2::/64 0xef28dd00 N 2279 sec 2 2684354561 1111::20 1111::10 192.168.5.0/24 192.168.1.0/24 0x3c0ccb00 N 922 sec 2 3758096386 1111::20 1111::10 fec0:5::/64 fec0:1::/64 0x23397400 N 880 sec 2 2481958963 2424::101 5050::100 0xe5eb0d00 N 1581 sec IPsec-NoVTIPacket dropped due to IPsec input missing or admin-down VTI interface VTI interface present on T0/T1 SR with mode as "vti" and we can find it by running "get logical-router <T0/T1 SR UUID> interfaces" edge1> get logical-router 220c92e1-343b-404f-b868-6c8365762d7b interfaces Logical Router UUID VRF LR-ID Name Type 220c92e1-343b-404f-b868-6c8365762d7b 2 3 SR-ServerT0_AS SERVICE_ROUTER_TIER0 Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable) Interface : cec70165-6e00-4ae8-93ef-b0338614a5e3 Ifuid : 295 Mode : vti Port-type : vti IP/Mask : 192.168.37.102/30;fec0:37::102/64(NA);fec0:37::50:56ff:fe01:400/64(NA);fe80::50:56ff:fe01:400/64(NA) Urpf-mode : PORT_CHECK Admin : up Op_state : up TX Interface Stats Stats(Tx)DescriptionActionBlockedPacket dropped due to interface being blocked/admin-down due to SR of the edge node is in standby stateSee RxFirewallPacket dropped due to firewallSee RxFrag-NeededPacket dropped due to the need of fragmentationDF bit is set but packet len is greater than than the MTU, check MTU configurationNo-neighborPacket dropped due to ARP failureThe neighbor IP has not reply the arp request, check the neighbor VM or routerNo-MemoryPacket dropped due to shortage of memoryPotential mbuf/memory leaking Edge1> get dataplane memory stats Mon Feb 12 2024 UTC 03:39:39.950 Memory Usage Available_entries : 1024 Available_entries_in_cache : 0 Cache_size_per_core : 128 Name : jumbo_mbuf_pool Size : 1024 No-Linked-PortPacket dropped due to failure in forwarding due to no linked peer lport Check lrouter port configuration, it should have linked lswitch port like nsx> get logical-router interface c32d0564-72f9-47c4-9d87-231eda4843ff Fri Jan 26 2024 PST 13:56:49.745 interface : c32d0564-72f9-47c4-9d87-231eda4843ff LS port : 18ba9b76-23d9-4963-90e8-24b9d595a5fb Rate-LimitPacket dropped due to egress traffic rate higher than configured QOS (traffic contract) Check if egress traffic to the lrouter is much higher than the QOS config. You can find the QOS configured using the below CLI, Edge1> get logical-router 558ba208-8ae9-4b74-a379-55d94e3577ed qos-config Mon Jan 29 2024 UTC 06:16:45.835 Logical Router UUID : 558ba208-8ae9-4b74-a379-55d94e3577ed Direction : Egress Rate (Mbps) : 1 Burst (bytes) : 1 IPsecPacket dropped during IPsec output processing Aggregated Tx drop counter for IPsec related drop, further check IPsec tunnel stats for particular drop reason using following command "get ipsecvpn tunnel stats <IPsec tunnel/VTI UUID>" edge1> get ipsecvpn tunnel stats cec70165-6e00-4ae8-93ef-b0338614a5e3 Interface UID : 295 Interface UUID : cec70165-6e00-4ae8-93ef-b0338614a5e3 VTI UUID : cec70165-6e00-4ae8-93ef-b0338614a5e3 Stats Rx Pkts : 0 Tx Pkts : 0 Rx Bytes : 0 Tx Bytes : 0 Rx MSS Adjusted : 0 Tx MSS Adjusted : 0 Rx MSS Ignored : 0 Tx MSS Ignored : 0 Rx Drops : 0 Tx Drops : 0 Rx Drop Crypto Failure : 0 Tx Drop Crypto Failure : 0 Rx Drop Enqueue Failure : 0 Tx Drop Enqueue Failure : 0 Rx Drop State Mismatch : 0 Tx Drop State Mismatch : 0 Rx Drop Malformed : 0 Tx Drop Malformed : 0 Rx Drop Proto Not Supported : 0 Tx Drop Proto Not Supported : 0 Rx Drop Replay : 0 Tx Drop Seq Rollover : 0 Rx Drop Inner Malformed : 0 Tx Drop Fragmentation Needed : 0 Rx Drop Policy Nomatch : 0 Rekey Request Failure : 0 Rx Drop Auth Failure : 0 Rx Drop Zero Sequence Number : 0 v6 Stats Rx Pkts : 3236 Tx Pkts : 3184 Rx Bytes : 250276 Tx Bytes : 631712 Rx MSS Adjusted : 0 Tx MSS Adjusted : 0 Rx MSS Ignored : 0 Tx MSS Ignored : 0 Rx Drops : 0 Tx Drops : 0 Rx Drop Crypto Failure : 0 Tx Drop Crypto Failure : 0 Rx Drop Enqueue Failure : 0 Tx Drop Enqueue Failure : 0 Rx Drop State Mismatch : 0 Tx Drop State Mismatch : 0 Rx Drop Malformed : 0 Tx Drop Malformed : 0 Rx Drop Proto Not Supported : 0 Tx Drop Proto Not Supported : 0 Rx Drop Replay : 0 Tx Drop Seq Rollover : 0 Rx Drop Inner Malformed : 0 Tx Drop Fragmentation Needed : 0 Rx Drop Policy Nomatch : 0 Rekey Request Failure : 0 Rx Drop Auth Failure : 0 Rx Drop Zero Sequence Number : 0 ---------------------------------------------------------------------------------------------------- IPsec-NoSAPacket dropped due to missing IPsec outbound or egress security association (SA) Check Outbound and Inbound SAs installed by running below commands "get ipsecvpn sad" or "get ipsecvpn sad summary" edge1> get ipsecvpn sad summary Outbound SAs (6): ---------------------------------------------------------------------------------------------------------------------------- VRF Rule ID Src IP Dest IP Src Subnet Dest Subnet SPI NAT Rem Life ---------------------------------------------------------------------------------------------------------------------------- 2 1408217139 5050::100 2424::101 0xc6a6b127 N 1642 sec 2 334475315 5050::100 2424::101 0xcbe1de7e N 1581 sec 2 536870913 1111::10 1111::20 192.168.1.0/24 192.168.5.0/24 0x2e733800 N 922 sec 5 536870927 192.168.51.100 20.20.20.101 192.168.2.0/24 192.168.7.0/24 0xce11f2c1 N 2279 sec 2 1610612738 1111::10 1111::20 fec0:1::/64 fec0:5::/64 0x2786ae00 N 880 sec 5 1610612752 192.168.51.100 20.20.20.101 fec0:2::/64 fec0:7::/64 0xc02800f4 N 2279 sec Inbound SAs (6): ---------------------------------------------------------------------------------------------------------------------------- VRF Rule ID Src IP Dest IP Src Subnet Dest Subnet SPI NAT Rem Life ---------------------------------------------------------------------------------------------------------------------------- 5 2684354575 20.20.20.101 192.168.51.100 192.168.7.0/24 192.168.2.0/24 0xbf806100 N 2279 sec 2 3555700787 2424::101 5050::100 0xc3061f00 N 1642 sec 5 3758096400 20.20.20.101 192.168.51.100 fec0:7::/64 fec0:2::/64 0xef28dd00 N 2279 sec 2 2684354561 1111::20 1111::10 192.168.5.0/24 192.168.1.0/24 0x3c0ccb00 N 922 sec 2 3758096386 1111::20 1111::10 fec0:5::/64 fec0:1::/64 0x23397400 N 880 sec 2 2481958963 2424::101 5050::100 0xe5eb0d00 N 1581 sec IPsec-NoVTIPacket dropped due to IPsec input missing or down VTI interface VTI interface present on T0/T1 SR with mode as "vti" and we can find it by running "get logical-router <T0/T1 SR UUID> interfaces" edge1> get logical-router 220c92e1-343b-404f-b868-6c8365762d7b interfaces Logical Router UUID VRF LR-ID Name Type 220c92e1-343b-404f-b868-6c8365762d7b 2 3 SR-ServerT0_AS SERVICE_ROUTER_TIER0 Interfaces (IPv6 DAD Status A-DAD_Success, F-DAD_Duplicate, T-DAD_Tentative, U-DAD_Unavailable) Interface : cec70165-6e00-4ae8-93ef-b0338614a5e3 Ifuid : 295 Mode : vti Port-type : vti IP/Mask : 192.168.37.102/30;fec0:37::102/64(NA);fec0:37::50:56ff:fe01:400/64(NA);fe80::50:56ff:fe01:400/64(NA) Urpf-mode : PORT_CHECK Admin : up Op_state : up IPsec-Policy-ErrorPacket dropped due to missing rule for IPsec processing. SPD look-up failed. Check below command output there should not be any entry with tag "ipsec" edge1> get firewall ded3f395-bd88-4c11-8472-75e6b1422837 ike policy IKE policy count: 4 Rule ID : 536870913 Policy : out protocol any stateless from ip 192.168.1.0/24 to ip 192.168.5.0/24 secure keypolicy 00003400-2000-0000-2000-000100000000 tag 'ipsec' Rule ID : 1610612738 Policy : out protocol any stateless from ip fec0:1::/64 to ip fec0:5::/64 secure keypolicy 00003400-2000-0000-2000-000200000000 tag 'ipsec' IPsec-Policy-BlockPacket dropped due to IPsec rule with DROP as action Check below command output for tag "DROP" edge1> get firewall ded3f395-bd88-4c11-8472-75e6b1422837 ike policy IKE policy count: 4 Rule ID : 536870913 Policy : out protocol any stateless from ip 192.168.1.0/24 to ip 192.168.5.0/24 secure keypolicy 00003400-2000-0000-2000-000100000000 tag 'ipsec' Rule ID : 1610612738 Policy : out protocol any stateless from ip fec0:1::/64 to ip fec0:5::/64 secure keypolicy 00003400-2000-0000-2000-000200000000 tag 'ipsec'