Issue

What were you trying to do that didn't work? The qemu-kvm crashes when post-copy migrating the rt-VM with vhost-user interfaces for multi times Please provide the package NVR for which bug is seen: host: qemu-kvm-8.1.0-4.el9.x86_64 5.14.0-389.el9.x86_64+rt guest: 5.14.0-389.el9.x86_64+rt How reproducible: 100% Steps to reproduce 1. create a ovs-dpdk 2. start a rt-VM with three 1Q vhost-user interfaces 3. post-copy migrate the VM 4. repeat the migration multiple times In my case, the migration fails at the 3th times 5. check the migration job/qemu-kvm log On the source host: initiating migration qemu-kvm: ../softmmu/runstate.c:199: void runstate_set(RunState): Assertion `new_state < RUN_STATE__MAX' failed. shutting down, reason=crashed Expected results The qemu-kvm will not crash Actual results The qemu-kvm crashes Additional info: (1) the VM's qemu-kvm coredump info on the source host: # coredumpctl info PID: 6701 (qemu-kvm) UID: 0 (root) GID: 107 (qemu) Signal: 6 (ABRT) Timestamp: Tue 2023-11-28 04:07:24 EST (1h 8min ago) Command Line: /usr/libexec/qemu-kvm -name guest=rhel9.4,debug-threads=on -S -object $'{"qom-type":"secret","id":"masterKey0","format> Executable: /usr/libexec/qemu-kvm Control Group: /machine.slice/machine-qemu\x2d2\x2drhel9.4.scope/libvirt/emulator Unit: machine-qemu\x2d2\x2drhel9.4.scope Slice: machine.slice Boot ID: 2a7bcb0dc2b2436abbacb3aa393dfe7f Machine ID: 09f2cab20f914272bf3d34215f7fe2ab Hostname: dell-per740-03.lab.eng.pek2.redhat.com Storage: /var/lib/systemd/coredump/core.qemu-kvm.0.2a7bcb0dc2b2436abbacb3aa393dfe7f.6701.1701162444000000.zst (present) Size on Disk: 1.5M Message: Process 6701 (qemu-kvm) of user 0 dumped core. Stack trace of thread 6831: #0 0x00007f246e4e265c __pthread_kill_implementation (libc.so.6 + 0xa365c) #1 0x00007f246e493d06 raise (libc.so.6 + 0x54d06) #2 0x00007f246e4677f3 abort (libc.so.6 + 0x287f3) #3 0x00007f246e46771b __assert_fail_base.cold (libc.so.6 + 0x2871b) #4 0x00007f246e48cca6 __assert_fail (libc.so.6 + 0x4dca6) #5 0x00005637a610d8db runstate_set (qemu-kvm + 0x53b8db) #6 0x00005637a613d60c migration_thread (qemu-kvm + 0x56b60c) #7 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #8 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #9 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6720: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6721: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6722: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6723: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6724: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6725: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a60ffb9b qemu_wait_io_event (qemu-kvm + 0x52db9b) #4 0x00005637a634f161 kvm_vcpu_thread_fn (qemu-kvm + 0x77d161) #5 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #6 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #7 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6701: #0 0x00007f246e5819be ppoll (libc.so.6 + 0x1429be) #1 0x00005637a65d64f6 main_loop_wait (qemu-kvm + 0xa044f6) #2 0x00005637a610e637 qemu_main_loop (qemu-kvm + 0x53c637) #3 0x00005637a5f59cba qemu_default_main (qemu-kvm + 0x387cba) #4 0x00007f246e47eeb0 __libc_start_call_main (libc.so.6 + 0x3feb0) #5 0x00007f246e47ef60 __libc_start_main@@GLIBC_2.34 (libc.so.6 + 0x3ff60) #6 0x00005637a5f593d5 _start (qemu-kvm + 0x3873d5) Stack trace of thread 6706: #0 0x00007f246e47de5d syscall (libc.so.6 + 0x3ee5d) #1 0x00005637a65bcd4f qemu_event_wait (qemu-kvm + 0x9ead4f) #2 0x00005637a65cae49 call_rcu_thread (qemu-kvm + 0x9f8e49) #3 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #4 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #5 0x00007f246e47e450 __clone3 (libc.so.6 + 0x3f450) Stack trace of thread 6719: #0 0x00007f246e5818bf __poll (libc.so.6 + 0x1428bf) #1 0x00007f246e8751fc g_main_context_iterate.constprop.0 (libglib-2.0.so.0 + 0xaa1fc) #2 0x00007f246e81f5a3 g_main_loop_run (libglib-2.0.so.0 + 0x545a3) #3 0x00005637a63a9dbf iothread_run (qemu-kvm + 0x7d7dbf) #4 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #5 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #6 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) Stack trace of thread 6726: #0 0x00007f246e4dd4aa __futex_abstimed_wait_common (libc.so.6 + 0x9e4aa) #1 0x00007f246e4dfcb0 pthread_cond_wait@@GLIBC_2.3.2 (libc.so.6 + 0xa0cb0) #2 0x00005637a65bc5d6 qemu_cond_wait_impl (qemu-kvm + 0x9ea5d6) #3 0x00005637a5f96416 vnc_worker_thread.llvm.11853996392063472260 (qemu-kvm + 0x3c4416) #4 0x00005637a65bcfea qemu_thread_start.llvm.17279737596791441160 (qemu-kvm + 0x9eafea) #5 0x00007f246e4e0912 start_thread (libc.so.6 + 0xa1912) #6 0x00007f246e47e314 __clone (libc.so.6 + 0x3f314) ELF object binary architecture: AMD x86-64 (2) The VM xml can be as following: [1] http://10.73.72.41/log/bug/JIRA-17369/iommu/VM.xml [2] http://10.73.72.41/log/bug/JIRA-17369/non-iommu/VM.xml

Release Notes

No. of Comments

Resolution

Cannot Reproduce