Issue

Description of problem: "sockperf pingpong multicast pkey/vlan" test case consistently fails when libvma is tested on MLX4 IB0 devices. vma test results on rdma-virt-00/rdma-virt-01 & Beaker job J:7282087: 4.18.0-438.el8.x86_64, rdma-core-41.0-1.el8, mlx4, ib0, ConnectX-3 Pro & mlx4_0 Result | Status | Test ------------------------------------------------- PASS | 0 | sockperf pingpong multicast PASS | 0 | sockperf throughput multicast PASS | 0 | sockperf throughput unicast PASS | 0 | sockperf pingpong unicast PASS | 0 | sockperf (100 sockets) pingpong multicast PASS | 0 | sockperf (100 sockets) pingpong unicast FAIL | 1 | sockperf pingpong multicast pkey/vlan <<<============ PASS | 0 | sockperf pingpong unicast pkey/vlan Checking for failures and known issues: sockperf pingpong multicast pkey/vlan is NOT a known issue on any environment - consider filing a BZ +++++++++++++++++++++++++ This is a regression where, with RHEL-8.7.0, the same test passes. vma test results on rdma-virt-00/rdma-virt-01 & Beaker job J:7282212: 4.18.0-425.3.1.el8.x86_64, rdma-core-41.0-1.el8, mlx4, ib0, ConnectX-3 Pro & mlx4_0 Result | Status | Test ------------------------------------------------- PASS | 0 | sockperf pingpong multicast PASS | 0 | sockperf throughput multicast PASS | 0 | sockperf throughput unicast PASS | 0 | sockperf pingpong unicast PASS | 0 | sockperf (100 sockets) pingpong multicast PASS | 0 | sockperf (100 sockets) pingpong unicast PASS | 0 | sockperf pingpong multicast pkey/vlan <<<===== PASS | 0 | sockperf pingpong unicast pkey/vlan Checking for failures and known issues: no test failures Version-Release number of selected component (if applicable): Clients: rdma-virt-01 Servers: rdma-virt-00 DISTRO=RHEL-8.8.0-20221120.2 + [22-11-27 12:50:17] cat /etc/redhat-release Red Hat Enterprise Linux release 8.8 Beta (Ootpa) + [22-11-27 12:50:17] uname -a Linux rdma-virt-01.rdma.lab.eng.rdu2.redhat.com 4.18.0-438.el8.x86_64 #1 SMP Mon Nov 14 13:08:07 EST 2022 x86_64 x86_64 x86_64 GNU/Linux + [22-11-27 12:50:17] cat /proc/cmdline BOOT_IMAGE=(hd0,msdos1)/vmlinuz-4.18.0-438.el8.x86_64 root=UUID=7a33441c-1f35-4485-a8ce-fb7f97d8ec9b ro intel_idle.max_cstate=0 processor.max_cstate=0 intel_iommu=on iommu=on console=tty0 rd_NO_PLYMOUTH crashkernel=auto resume=UUID=38129101-96b6-4039-9d07-de4355860d19 console=ttyS1,115200n81 + [22-11-27 12:50:17] rpm -q rdma-core linux-firmware rdma-core-41.0-1.el8.x86_64 linux-firmware-20220726-110.git150864a4.el8.noarch + [22-11-27 12:50:17] tail /sys/class/infiniband/mlx4_0/fw_ver /sys/class/infiniband/mlx4_1/fw_ver ==> /sys/class/infiniband/mlx4_0/fw_ver <== 2.42.5000 ==> /sys/class/infiniband/mlx4_1/fw_ver <== 2.42.5000 + [22-11-27 12:50:17] lspci + [22-11-27 12:50:17] grep -i -e ethernet -e infiniband -e omni -e ConnectX 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme BCM5720 Gigabit Ethernet PCIe 04:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] 04:00.1 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.2 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.3 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.4 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.5 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.6 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 04:00.7 Network controller: Mellanox Technologies MT27500/MT27520 Family [ConnectX-3/ConnectX-3 Pro Virtual Function] 06:00.0 Network controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] + [22-11-27 12:50:18] rpm -q libvma libvma-9.6.4-1.el8.x86_64 How reproducible: 100% Steps to Reproduce: On both server and client hosts: + [22-11-27 12:58:15] export SERVER_IP2=172.31.2.200 + [22-11-27 12:58:15] SERVER_IP2=172.31.2.200 1. Get both RDMA server & client hosts ready for libvma test with current sockperf, with RHEL-8.8 build as specified above on MLX4 IB0 devices 2. On server host, issue the following commands LD_PRELOAD=libvma.so timeout --preserve-status --kill-after=5m 3m sockperf server -i 172.31.2.200 3. On the client hosts, issue the following command LD_PRELOAD=libvma.so timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 Actual results: [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m+ [22-11-27 13:00:53] LD_PRELOAD=libvma.so [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m+ [22-11-27 13:00:53] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0msockperf: [2;35m== version #3.10-0.git5ebd327da983 == [0m sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s) [ 0] IP = 172.31.2.200 PORT = 11111 # UDP sockperf: Warmup stage (sending a few dummy messages)... sockperf: Starting test... sockperf: Test end (interrupted by timer) sockperf: Test ended sockperf: No messages were received from the server. Is the server down? + [22-11-27 13:01:07] result=0 + [22-11-27 13:01:07] '[' 0 -ne 0 ']' + [22-11-27 13:01:07] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt + [22-11-27 13:01:07] return 1 Expected results: On RHEL-8.7.0, the above same testcase result look like the following: [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m+ [22-11-27 16:42:52] LD_PRELOAD=libvma.so [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: date ++ [%y-%m-%d %H:%M:%S] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m+ [22-11-27 16:42:52] timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: timeout --preserve-status --kill-after=5m 3m sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: VMA_VERSION: 9.6.4-1 Release built on Aug 17 2022 14:23:56 [0m[0m VMA INFO: Cmd Line: sockperf pp -i 172.31.2.200 -t 10 --msg-size=1472 [0m[0m VMA INFO: --------------------------------------------------------------------------- [0m[0m VMA INFO: Log Level INFO [VMA_TRACELEVEL] [0m[0m VMA INFO: --------------------------------------------------------------------------- [0msockperf: [2;35m== version #3.10-0.git5ebd327da983 == [0m sockperf[CLIENT] send on:sockperf: using recvfrom() to block on socket(s) [ 0] IP = 172.31.2.200 PORT = 11111 # UDP sockperf: Warmup stage (sending a few dummy messages)... sockperf: Starting test... sockperf: Test end (interrupted by timer) sockperf: Test ended sockperf: [Total Run] RunTime=10.000 sec; Warm up time=400 msec; SentMessages=1785444; ReceivedMessages=1785443 sockperf: ========= Printing statistics for Server No: 0 sockperf: [Valid Duration] RunTime=9.550 sec; SentMessages=1711702; ReceivedMessages=1711702 sockperf: [2;35m====> avg-latency=2.772 (std-dev=0.286, mean-ad=0.081, median-ad=0.036, siqr=0.028, cv=0.103, std-error=0.000, 99.0% ci=[2.771, 2.773])[0m sockperf: # dropped messages = 0; # duplicated messages = 0; # out-of-order messages = 0 sockperf: Summary: Latency is 2.772 usec sockperf: [2;35mTotal 1711702 observations[0m; each percentile contains 17117.02 observations sockperf: ---> <MAX> observation = 81.937 sockperf: ---> percentile 99.999 = 55.007 sockperf: ---> percentile 99.990 = 4.831 sockperf: ---> percentile 99.900 = 4.301 sockperf: ---> percentile 99.000 = 3.303 sockperf: ---> percentile 90.000 = 2.901 sockperf: ---> percentile 75.000 = 2.763 sockperf: ---> percentile 50.000 = 2.728 sockperf: ---> percentile 25.000 = 2.706 sockperf: ---> <MIN> observation = 2.621 + [22-11-27 16:43:06] result=0 + [22-11-27 16:43:06] '[' 0 -ne 0 ']' + [22-11-27 16:43:06] grep -qi -e ' error ' -e 'no messages were received' /tmp/vma.txt + [22-11-27 16:43:06] return 0 + [22-11-27 16:43:06] RQA_check_result -r 0 -t 'sockperf pingpong multicast pkey/vlan' Additional info:

Release Notes

No. of Comments

Resolution

Won't Do