Issue

What were you trying to do that didn't work? First installed 4.18.0-372.9.1.el8.x86_64 as base OS Then using Ansible, upgrade to 4.18.0-372.32.1.rt7.189.el8_6.x86_64: ansible-ansible.legacy.dnf[14000]: Invoked with name=['kernel-rt-4.18.0-372.32.1.rt7.189.el8_6', ' kernel-rt-modules-extra-4.18.0-372.32.1.rt7.189.el8_6'] allow_downgrade=True autoremove=False bugfix=False cacheonly=False disable_gpg_check=False disable_plugin=[] disablerepo=[] download_only=False enable_plugin=[] enablerepo=[] exclude=[] installroot=/ install_repoquery=True install_weak_deps=True security=False skip_broken=False update_cache=False update_only=False validate_certs=True sslverify=True lock_timeout=30 allowerasing=False nobest=False use_backend=auto conf_file=None disable_excludes=None download_dir=None list=None releasever=None state=None ansible-ansible.legacy.command[37459]: Invoked with _raw_params=grubby --args="default_hugepagesz=1G hugepagesz=1G hugepages=40 isolcpus=2-31 intel_pstate=disable nosoftlockup nohz=on nohz_full=2-31 rcu_nocbs=2-31 coredump_filter=0x73 intel_iommu=on rcutree.kthread_prio=90 rcu_nocb_poll pci=assign-busses pci=realloc efi=runtime" --update-kernel /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 _uses_shell=False expand_argument_vars=True stdin_add_newline=True strip_empty_ends=True argv=None chdir=None executable=None creates=None removes=None stdin=None Then reboot. 1 out of 10 times, the second menu entry (index=1) was used which booted into the old kernel 4.18.0-372.9.1.el8.x86_64 After this incorrect boot, verify the following (using the instruction from https://access.redhat.com/solutions/4326431): uname -a Linux B04R7V3 4.18.0-372.9.1.el8.x86_64 #1 SMP Fri Apr 15 22:12:19 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux grubby --default-kernel /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 grubby --default-index 0 grubby --info=ALL | grep title title="Red Hat Enterprise Linux (4.18.0-372.32.1.rt7.189.el8_6.x86_64) 8.6 (Ootpa)" title="Red Hat Enterprise Linux (4.18.0-372.9.1.el8.x86_64) 8.6 (Ootpa)" title="Red Hat Enterprise Linux (0-rescue-02389a0e819b4ed6a39e95091763e406) 8.6 (Ootpa)" grep ^id /boot/loader/entries/* /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-0-rescue.conf:id rhel-20220416023408-0-rescue-02389a0e819b4ed6a39e95091763e406 /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64.conf:id rhel-20221007175936-4.18.0-372.32.1.rt7.189.el8_6.x86_64 /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-4.18.0-372.9.1.el8.x86_64.conf:id rhel-20220416023408-4.18.0-372.9.1.el8.x86_64 grub2-editenv /boot/grub2/grubenv list saved_entry=02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 grub2-editenv /boot/efi/EFI/redhat/grubenv list saved_entry=02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 cat /etc/sysconfig/kernel | grep -i DEFAULTKERNEL # DEFAULTKERNEL specifies the default kernel package type DEFAULTKERNEL=kernel-core ls -l /boot/vmlinuz-* -rwxr-xr-x. 1 root root 10460528 Jan 20 04:55 /boot/vmlinuz-0-rescue-02389a0e819b4ed6a39e95091763e406 -rwxr-xr-x. 1 root root 10513776 Oct  7  2022 /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 -rwxr-xr-x. 1 root root 10460528 Apr 15  2022 /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 grubby --info=ALL index=0 kernel="/boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet $tuned_params default_hugepagesz=1G hugepagesz=1G hugepages=40 isolcpus=2-31 intel_pstate=disable nosoftlockup nohz=on nohz_full=2-31 rcu_nocbs=2-31 coredump_filter=0x73 intel_iommu=on rcutree.kthread_prio=90 rcu_nocb_poll pci=assign-busses pci=realloc efi=runtime" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-4.18.0-372.32.1.rt7.189.el8_6.x86_64.img $tuned_initrd" title="Red Hat Enterprise Linux (4.18.0-372.32.1.rt7.189.el8_6.x86_64) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64" index=1 kernel="/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet $tuned_params" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-4.18.0-372.9.1.el8.x86_64.img $tuned_initrd" title="Red Hat Enterprise Linux (4.18.0-372.9.1.el8.x86_64) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-4.18.0-372.9.1.el8.x86_64" index=2 kernel="/boot/vmlinuz-0-rescue-02389a0e819b4ed6a39e95091763e406" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-0-rescue-02389a0e819b4ed6a39e95091763e406.img" title="Red Hat Enterprise Linux (0-rescue-02389a0e819b4ed6a39e95091763e406) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-0-rescue" Then reboot again without changing any grub configuration, it chose menu entry index 1 again as default: If reboot again and choose index 0 manually (4.18.0-372.32.1.rt7.189.el8_6.x86_64), after boot, verify the following: uname -a Linux B04R7V3 4.18.0-372.32.1.rt7.189.el8_6.x86_64 #1 SMP PREEMPT_RT Fri Oct 7 13:32:56 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux grubby --default-kernel /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 grubby --default-index 0 grubby --info=ALL | grep title title="Red Hat Enterprise Linux (4.18.0-372.32.1.rt7.189.el8_6.x86_64) 8.6 (Ootpa)" title="Red Hat Enterprise Linux (4.18.0-372.9.1.el8.x86_64) 8.6 (Ootpa)" title="Red Hat Enterprise Linux (0-rescue-02389a0e819b4ed6a39e95091763e406) 8.6 (Ootpa)" grep ^id /boot/loader/entries/* /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-0-rescue.conf:id rhel-20220416023408-0-rescue-02389a0e819b4ed6a39e95091763e406 /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64.conf:id rhel-20221007175936-4.18.0-372.32.1.rt7.189.el8_6.x86_64 /boot/loader/entries/02389a0e819b4ed6a39e95091763e406-4.18.0-372.9.1.el8.x86_64.conf:id rhel-20220416023408-4.18.0-372.9.1.el8.x86_64 grub2-editenv /boot/grub2/grubenv list saved_entry=02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 grub2-editenv /boot/efi/EFI/redhat/grubenv list saved_entry=02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 cat /etc/sysconfig/kernel | grep -i DEFAULTKERNEL # DEFAULTKERNEL specifies the default kernel package type DEFAULTKERNEL=kernel-core ls -l /boot/vmlinuz-* -rwxr-xr-x. 1 root root 10460528 Jan 20 04:55 /boot/vmlinuz-0-rescue-02389a0e819b4ed6a39e95091763e406 -rwxr-xr-x. 1 root root 10513776 Oct  7  2022 /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 -rwxr-xr-x. 1 root root 10460528 Apr 15  2022 /boot/vmlinuz-4.18.0-372.9.1.el8.x86_64 grubby --info=ALL index=0 kernel="/boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet $tuned_params default_hugepagesz=1G hugepagesz=1G hugepages=40 isolcpus=2-31 intel_pstate=disable nosoftlockup nohz=on nohz_full=2-31 rcu_nocbs=2-31 coredump_filter=0x73 intel_iommu=on rcutree.kthread_prio=90 rcu_nocb_poll pci=assign-busses pci=realloc efi=runtime" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-4.18.0-372.32.1.rt7.189.el8_6.x86_64.img $tuned_initrd" title="Red Hat Enterprise Linux (4.18.0-372.32.1.rt7.189.el8_6.x86_64) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-4.18.0-372.32.1.rt7.189.el8_6.x86_64" index=1 kernel="/boot/vmlinuz-4.18.0-372.9.1.el8.x86_64" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet $tuned_params" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-4.18.0-372.9.1.el8.x86_64.img $tuned_initrd" title="Red Hat Enterprise Linux (4.18.0-372.9.1.el8.x86_64) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-4.18.0-372.9.1.el8.x86_64" index=2 kernel="/boot/vmlinuz-0-rescue-02389a0e819b4ed6a39e95091763e406" args="ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet" root="/dev/mapper/rhel-root" initrd="/boot/initramfs-0-rescue-02389a0e819b4ed6a39e95091763e406.img" title="Red Hat Enterprise Linux (0-rescue-02389a0e819b4ed6a39e95091763e406) 8.6 (Ootpa)" id="02389a0e819b4ed6a39e95091763e406-0-rescue" Then reboot again without changing any grub configuration, the default is index=1 again (4.18.0-372.9.1.el8.x86_64) and boot into the old kernel ========================================================== compare with a setup in which this issue didn't occur (9 out of 10 times): the printout are identical except the following: uname -a Linux 704R7V3 4.18.0-372.32.1.rt7.189.el8_6.x86_64 #1 SMP PREEMPT_RT Fri Oct 7 13:32:56 EDT 2022 x86_64 x86_64 x86_64 GNU/Linux grubby --default-kernel /boot/vmlinuz-4.18.0-372.32.1.rt7.189.el8_6.x86_64 In the working setup, uname and grubby default are the same. They also differ from the below printout: grub2-editenv /boot/grub2/grubenv list saved_entry=b651008b393b478da50b08dad8ac1d89-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 boot_indeterminate=0 grub2-editenv /boot/efi/EFI/redhat/grubenv list saved_entry=b651008b393b478da50b08dad8ac1d89-4.18.0-372.32.1.rt7.189.el8_6.x86_64 kernelopts=root=/dev/mapper/rhel-root ro console=tty1 rd.lvm.lv=rhel/root rhgb quiet boot_success=1 boot_indeterminate=0 In the working setup `boot_indeterminate=0` exists but not in the setup in which it boot into the old kernel.

Release Notes

No. of Comments

Resolution

Cannot Reproduce