Issue

Description of problem: The domain with vhost-user interface + iommu throws "call trace" when running the netperf tests Version-Release number of selected component (if applicable): qemu-kvm-8.0.0-4.el9.x86_64 5.14.0-323.el9.x86_64 dpdk-22.11-3.el9_2.x86_64 openvswitch3.1-3.1.0-28.el9fdp.x86_64 How reproducible: 100% Steps to Reproduce: 1. setup the host kernel option, like CPU isolation,huge-page, iommu # grubby --args="iommu=pt intel_iommu=on default_hugepagesz=1G" --update-kernel=`grubby --default-kernel` # echo "isolated_cores=2,4,6,8,10,12,14,16,18,20,22,24,26,28,30,31,29,27,25,23,21,19,17,15,13,11" >> /etc/tuned/cpu-partitioning-variables.conf tuned-adm profile cpu-partitioning # reboot 2. start a ovs-dpdk on the host # echo 20 > /sys/devices/system/node/node0/hugepages/hugepages-1048576kB/nr_hugepages # echo 20 > /sys/devices/system/node/node1/hugepages/hugepages-1048576kB/nr_hugepages # modprobe vfio # modprobe vfio-pci # dpdk-devbind.py --bind=vfio-pci 0000:5e:00.0 # dpdk-devbind.py --bind=vfio-pci 0000:5e:00.1 ... # ovs-vsctl get Open_vSwitch . other_config {dpdk-init="true", dpdk-lcore-mask="0x2", dpdk-socket-mem="1024,1024", pmd-cpu-mask="0x15554", vhost-iommu-support="true"} # ovs-vsctl show 1e271d29-308d-4201-be11-d898617cc592 Bridge ovsbr0 datapath_type: netdev Port ovsbr0 Interface ovsbr0 type: internal Port dpdk0 Interface dpdk0 type: dpdk options: {dpdk-devargs="0000:5e:00.0", n_rxq="2", n_txq="2"} Port vhost-user0 Interface vhost-user0 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser0.sock"} Bridge ovsbr1 datapath_type: netdev Port ovsbr1 Interface ovsbr1 type: internal Port dpdk1 Interface dpdk1 type: dpdk options: {dpdk-devargs="0000:5e:00.1", n_rxq="2", n_txq="2"} Port vhost-user1 Interface vhost-user1 type: dpdkvhostuserclient options: {vhost-server-path="/tmp/vhostuser1.sock"} 3. start a nfv virt domain with iommu device and vhost-user interfaces <interface type='vhostuser'> <mac address='18:66:da:5f:dd:22'/> <source type='unix' path='/tmp/vhostuser0.sock' mode='server'/> <target dev='vhost-user0'/> <model type='virtio'/> <driver name='vhost' queues='2' rx_queue_size='1024' iommu='on' ats='on'/> <alias name='net1'/> </interface> <iommu model='intel'> <driver intremap='on' caching_mode='on' iotlb='on'/> </iommu> 4. setup the kernel option in the domain # grubby --args="iommu=pt intel_iommu=on default_hugepagesz=1G" --update-kernel=`grubby --default-kernel` # echo "isolated_cores=1,2,3,4,5" >> /etc/tuned/cpu-partitioning-variables.conf # tuned-adm profile cpu-partitioning # reboot 5. run the netperf tests between the domain clinet and host server (5.1) The host is the netperf server # # ip addr add 192.168.1.3/24 dev ens3f1 # netserver Starting netserver with host 'IN(6)ADDR_ANY' port '12865' and family AF_UNSPEC (5.2)The domain is the netperf client: # ip addr add 192.168.1.2/24 dev enp6s0 <-- the domain can ping the 192.168.1.3 successfully but with some package lost # netperf -H 192.168.1.3/24 6. check the domain dmesg # dmesg [ 4802.234530] ------------[ cut here ]------------ [ 4802.234532] NETDEV WATCHDOG: enp6s0 (virtio_net): transmit queue 0 timed out [ 4802.234549] WARNING: CPU: 0 PID: 0 at net/sched/sch_generic.c:525 dev_watchdog+0x1f9/0x200 [ 4802.236690] Modules linked in: intel_rapl_msr intel_rapl_common isst_if_common nfit libnvdimm kvm_intel kvm nft_fib_inet nft_fib_ipv4 nft_fib_ipv6 nft_fib nft_reject_inet nf_reject_ipv4 nf_reject_ipv6 nft_reject nft_ct nft_chain_nat nf_nat nf_conntrack nf_defrag_ipv6 nf_defrag_ipv4 bridge ip_set stp llc iTCO_wdt rfkill iTCO_vendor_support nf_tables irqbypass nfnetlink rapl virtio_balloon i2c_i801 i2c_smbus lpc_ich qrtr pcspkr vfat fat drm fuse xfs libcrc32c ahci libahci nvme_tcp nvme_fabrics nvme libata nvme_core nvme_common t10_pi crct10dif_pclmul crc32_pclmul crc32c_intel virtio_net ghash_clmulni_intel net_failover virtio_blk failover serio_raw sunrpc dm_mirror dm_region_hash dm_log dm_mod [ 4802.243011] CPU: 0 PID: 0 Comm: swapper/0 Kdump: loaded Not tainted 5.14.0-323.el9.x86_64 #1 [ 4802.243900] Hardware name: Red Hat KVM/RHEL, BIOS edk2-20230301gitf80f052277c8-5.el9 03/01/2023 [ 4802.244809] RIP: 0010:dev_watchdog+0x1f9/0x200 [ 4802.245284] Code: 00 e9 40 ff ff ff 48 89 ef c6 05 03 af 7a 01 01 e8 3c c5 fa ff 44 89 e9 48 89 ee 48 c7 c7 a0 b1 6d 97 48 89 c2 e8 17 82 77 ff <0f> 0b e9 22 ff ff ff 0f 1f 44 00 00 55 53 48 89 fb 48 8b 6f 18 0f [ 4802.247210] RSP: 0018:ffffb32980003eb0 EFLAGS: 00010286 [ 4802.247766] RAX: 0000000000000000 RBX: ffff99428b8ff488 RCX: 0000000000000027 [ 4802.248511] RDX: 0000000000000027 RSI: ffffffff97e67460 RDI: ffff994337c1f8c8 [ 4802.249262] RBP: ffff99428b8ff000 R08: ffff994337c1f8c0 R09: 0000000000000000 [ 4802.250016] R10: ffffffffffffffff R11: ffffffff98b6f070 R12: ffff99428b8ff3dc [ 4802.250766] R13: 0000000000000000 R14: ffffffff96b7e5b0 R15: ffffb32980003f08 [ 4802.251516] FS: 0000000000000000(0000) GS:ffff994337c00000(0000) knlGS:0000000000000000 [ 4802.252364] CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033 [ 4802.252977] CR2: 00007ffe7a749000 CR3: 0000000101d54004 CR4: 0000000000770ef0 [ 4802.253732] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000 [ 4802.254479] DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400 [ 4802.255230] PKRU: 55555554 [ 4802.255532] Call Trace: [ 4802.255805] <IRQ> [ 4802.256031] ? pfifo_fast_change_tx_queue_len+0x70/0x70 [ 4802.256586] call_timer_fn+0x24/0x130 [ 4802.256986] __run_timers.part.0+0x1ee/0x280 [ 4802.257444] ? enqueue_hrtimer+0x2f/0x80 [ 4802.257870] ? __hrtimer_run_queues+0x159/0x2c0 [ 4802.258358] run_timer_softirq+0x26/0x50 [ 4802.258785] __do_softirq+0xc7/0x2ac [ 4802.259173] __irq_exit_rcu+0xb9/0xf0 [ 4802.259573] sysvec_apic_timer_interrupt+0x72/0x90 [ 4802.260084] </IRQ> [ 4802.260318] <TASK> [ 4802.260559] asm_sysvec_apic_timer_interrupt+0x16/0x20 [ 4802.261103] RIP: 0010:default_idle+0x10/0x20 [ 4802.261571] Code: 8b 04 25 40 ef 01 00 f0 80 60 02 df c3 cc cc cc cc 0f ae 38 eb bb 0f 1f 40 00 0f 1f 44 00 00 66 90 0f 00 2d be da 47 00 fb f4 <c3> cc cc cc cc cc cc cc cc cc cc cc cc cc cc cc 0f 1f 44 00 00 65 [ 4802.263496] RSP: 0018:ffffffff97e03ea8 EFLAGS: 00000252 [ 4802.264050] RAX: ffffffff96d8d320 RBX: ffffffff97e1a940 RCX: 0000000000000000 [ 4802.264803] RDX: 4000000000000000 RSI: ffff994337c22b20 RDI: 000000000497eebc [ 4802.265554] RBP: 0000000000000000 R08: 0000045e163d1cbb R09: ffff9941d6202400 [ 4802.266301] R10: 0000000000020604 R11: 0000000000000000 R12: 0000000000000000 [ 4802.267054] R13: 000000006dc53d18 R14: 000000006d3c47a8 R15: 000000006d3c47b0 [ 4802.267810] ? mwait_idle+0x70/0x70 [ 4802.268189] default_idle_call+0x33/0xe0 [ 4802.268615] cpuidle_idle_call+0x125/0x160 [ 4802.269051] ? kvm_sched_clock_read+0x14/0x30 [ 4802.269519] do_idle+0x78/0xe0 [ 4802.269891] cpu_startup_entry+0x19/0x20 [ 4802.270311] rest_init+0xca/0xd0 [ 4802.270671] arch_call_rest_init+0xa/0x14 [ 4802.271099] start_kernel+0x4a3/0x4c2 [ 4802.271495] secondary_startup_64_no_verify+0xe5/0xeb [ 4802.272037] </TASK> [ 4802.272279] ---[ end trace 87fb221169225dfd ]-- Besides above "Call Trace" , the domain will keep throwing the info like" virtio_net virtio3 enp6s0: TX timeout on queue: 0, sq: output.0, vq: 0x1, name: output.0, 7820000 usecs ago" 7. run the ping tests # ping 192.168.1.3 PING 192.168.1.3 (192.168.1.3) 56(84) bytes of data. ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ping: sendmsg: No buffer space available ... Actual results: The domain with vhost-user interface + iommu throws "call trace" when running the netperf tests Expected results: No Call Trace Additional info:

Release Notes

No. of Comments

Resolution

Done