Symptoms

An intermittent issue regarding high latency is seen when attempting to perform an ICMP ping from a client device to the gateway, generally the IP of the VLAN interface. This issue can still be seen even when directly connected to a gateway switch that has little to no production traffic. There has been some customer concern that this reflects an impact on normal production traffic. Issue seen across all S-series platforms running OS9 firmware. Test scenario for ICMP test (No production traffic) Topology Configuration PC1: 192.168.10.100/24, Gateway: 192.168.10.1/24, VLAN10 PC2: 192.168.10.200/24, Gateway: 192.168.10.1/24, VLAN10 or PC2: 192.168.20.200/24, Gateway: 192.168.20.1/24, VLAN20 Pings between client devices are not generally affected, so not all of the results of testing are listed here. This issue is only seen when pinging from a client device to a gateway IP on a switch. PC1 ping switch gateway (directly connected) Test 1-6; same result seen, regardless of topology Pinging 192.168.10.1 with 32 bytes of data: Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=47ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=5ms TTL=255 Reply from 192.168.10.1: bytes=32 time=1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=32ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=6ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time=40ms TTL=255 <------ high latency ! Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 Reply from 192.168.10.1: bytes=32 time<1ms TTL=255 …… >>> (truncated) Ping statistics for 192.168.10.1: Packets: Sent = 8280，Received = 8280，Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms，Maximum = 55ms，Average = 1ms Ping between two PCs, transit traffic (Layer2, in same and different vlan's) (Test 1-3) Pinging 192.168.10.200 with 32 bytes of data: Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 …… >>> (truncated), …… Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time<1ms TTL=128 Reply from 192.168.10.200: bytes=32 time=1ms TTL=128 Ping statistics for 192.168.10.200: Packets: Sent = 1233，Received = 1233，Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms，Maximum = 2ms，Average = 0ms Ping between two PCs, transit traffic (Layer3, in two vlans) (Tests 4-6) Pinging 192.168.20.200 with 32 bytes of data: Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 …… >>> (truncated), …… Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Reply from 192.168.20.200: bytes=32 time<1ms TTL=127 Reply from 192.168.20.200: bytes=32 time=1ms TTL=127 Ping statistics for 192.168.20.200: Packets: Sent = 808，Received = 808，Lost = 0 (0% Loss) Approximate round trip times in milli-seconds: Minimum = 0ms，Maximum = 2ms，Average = 0ms Explanation of Results It has been confirmed with Dell EMC Engineering that this is normal and expected behavior due to CPU scheduling and queuing. ICMP testing should only be used as a means to test reachability, rather than latency or loss due to this behavior. ICMP traffic latency is not a reflection of impact to normal production traffic. This behavior can also be exacerbated by a busy CPU - i.e. for a switch that is in full production, with a higher amount of traffic on it.

Cause

Resolution