OPERATIONAL DEFECT DATABASE
...
...
Multiple VM reporting high TX drops in vROP on NSX-T segments.No impact reported due to the packet dropsViewing VM port stats using vsish, we see drops: cat /net/portsets/DvsPortset-1/ports/67109164/inputStats DVFILTER_VNIC_IN_GUEST <vmware-sfw:0x4311fee63512> pktsStarted:53880402 pktsPassed:50530971 pktsDropped:3349431 pktsFiltered:0 pktsQueued:0 pktsFaulted:0 pktsInjected:0 pktErrors:0 cat /net/portsets/DvsPortset-0/ports/67109164/vmxnet3/txSummary stats of a vmxnet3 vNIC tx queue { generation:11223 pkts tx ok:52789244 bytes tx ok:9112557440 TSO pkts tx ok:571801 TSO bytes tx ok:2392954565 unicast pkts tx ok:52789237 unicast bytes tx ok:9112557146 multicast pkts tx ok:0 multicast bytes tx ok:0 broadcast pkts tx ok:7 broadcast bytes tx ok:294 pkts tx failure:0 pkts discarded:0 error when copying hdrs:0 tso header errors:0 pkt allocation failures:0 # of times a tx queue is stopped:0 failed to map some guest buffers:0 cat /net/portsets/DvsPortset-0/ports/67109164/stats packet stats { pktsTx:1170369462 pktsTxMulticast:0 pktsTxBroadcast:26 pktsRx:1223815657 pktsRxMulticast:0 pktsRxBroadcast:1163405 droppedTx:161590057 droppedRx:82201 Checking the DFW filter of the VM using summarize-dvfilter and vsipioctl #summarize-dvfilter world 72383945 vmm0:vx-4941097172 vcUuid:'50 0b 64 09 bc 0e 73 ba-61 b1 89 8c 9b a5 80 dc' port 67109164 vx-4941097172.eth0 vNic slot 2 name: nic-72383945-eth0-vmware-sfw.2 agentName: vmware-sfw state: IOChain Attached vmState: Attached failurePolicy: failClosed serviceVMID: 2 filter source: Dynamic Filter Creation moduleName: nsxt-vsip-19761813 #vsipioctl getfilters -s Filter Name : nic-72383945-eth0-vmware-sfw.2 VM UUID : 50 0b 64 09 bc 0e 73 ba-61 b1 89 8c 9b a5 80 dc VNIC Index : 0 VNIC UUID : 500b6409-bc0e-73ba-61b1-898c9ba580dc.000 VIF ID : c53ca8d1-d903-4f4a-9d4a-0245aa846c83 LSP ID : c5d61074-3ca1-42ef-9c4a-c5b0809ec81b Service Profile : --NOT SET-- Filter Rule Config : configured Filter Hash : 20132 Basic Stats: From Switch To Switch Total rx from dvfilter : 1199764347 1254357160 Total packets : 1199715455 1171952477 Total time (us) : 2020322128 12736318215 Rate (us/pkt) : 1 10 Rate (pkt/s) : 593823 92016 #vsipioctl getfilterstat -f nic-72383945-eth0-vmware-sfw.2 PACKETS IN OUT ------- -- --- v4 pass: 1198259988 1171938619 v4 drop: 336 18400 v4 reject: 17230 82386448 v6 pass: 0 0 v6 drop: 0 0 BYTES IN OUT ----- -- --- v4 pass: 208158739203 200721582402 v4 drop: 19520 956800 v4 reject: 641217 4943255924 v6 pass: 0 0 v6 drop: 0 0 DROP REASON ----------- src-limit: 211 strict no syn: 18736 3wh error: 7 FILTER INFO ----------- sessions: 2350916 flags: 0xe46 states: 615 rules: 321 table count: 174 filter version: 1100 ruleset gen: 1542067 hash: 20132 last purge: 6458993 fprn alloc err: 0
From above we can see a few packets are dropped due to not finding a SYN packet at the start of a new TCP flow, but this is not the issue here.In the getfilterstat results for the filter, we can see a lot of transmitted packets are getting rejected: PACKETS IN OUT ------- -- --- v4 pass: 1198259988 1171938619 v4 drop: 336 18400 v4 reject: 17230 82386448 If we check the rules, we see there is one rule with a similar amount of out packets getting rejected: #vsipioctl getrules -f nic-72383945-eth0-vmware-sfw.2 -s rule 2 at 35, 396409 evals, 82402926 hits, 235 sessions, in 17230 out 82386444 pkts, in 641217 out 4943255740 bytes The issue here is that the transmitted packet did not match any flows and eventually hit the default reject rule, which caused the counter to increment.
This is expected behaviour, as the traffic did not match any rule and the default rule was reject.To help narrow the issue down, you can enable logging on the default rule, then check the firewall logs on the ESXi host for the VM reporting packet drops: /var/run/log/dfwpktlogs.log Check the traffic flow for rejected rules and ensure they are correctly rejected and review traffic flow from the VM.
Make sure all traffic flows are matched with a rule.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
