Issue

Description of problem: When the system is booting with the NFS dracut modules, e.g. when using the generic initramfs (due to presence of dracut-config-generic package), the following ordering cycle is seen at shutdown: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- After much digging, I was able to find out that the reason was having an issue with /var/lib/nfs/rpc_pipefs mount: the unit has the following ordering dependencies (as seen by "systemd-analyze dump"): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Before: local-fs.target (origin-mountinfo-implicit) Before: rpc_pipefs.target (destination-file) Before: umount.target (origin-mountinfo-default) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- whereas it should not and just have those: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Before: rpc_pipefs.target (destination-file) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Additionally (but unrelated), in such case, we can see the mount is not correct, it is mounted as: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- rpc_pipefs on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- whereas it should be mounted as (see What): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime) -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- The root cause is the NFS dracut module: 1. firstly mounting with What == rpc_pipefs instead of What == sunrpc (this causes the discrepancy in mount parameters): -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- diff -u /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh.orig 2022-09-26 13:34:11.416891907 +0200 +++ /usr/lib/dracut/modules.d/95nfs/nfs-start-rpc.sh 2022-09-26 13:34:19.152877777 +0200 @@ -4,7 +4,7 @@ if modprobe sunrpc || strstr "$(cat /proc/filesystems)" rpc_pipefs; then [ ! -d /var/lib/nfs/rpc_pipefs/nfs ] && \ mount -t rpc_pipefs rpc_pipefs /var/lib/nfs/rpc_pipefs + mount -t rpc_pipefs sunrpc /var/lib/nfs/rpc_pipefs Start rpcbind or rpcbind FIXME occasionally saw 'rpcbind: fork failed: No such device' – why? 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- 2. secondly the cleanup doing a bind-mount of /var/lib/nfs/rpc_pipefs to /sysroot/var/lib/nfs/rpc_pipefs, which leads to having the unexpected "Before=local-fs.target umount.target" dependencies and "origin-mountinfo-implicit/origin-mountinfo-default" -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- diff -u /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh.orig 2022-09-26 13:43:19.441194953 +0200 +++ /usr/lib/dracut/modules.d/95nfs/nfsroot-cleanup.sh 2022-09-26 13:43:59.199659403 +0200 @@ -21,7 +21,7 @@ if [ -d $NEWROOT/$rpcpipefspath ]; then mount --move does not seem to work??? mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath + #mount --bind /var/lib/nfs/rpc_pipefs $NEWROOT/$rpcpipefspath umount /var/lib/nfs/rpc_pipefs 2>/dev/null else umount /var/lib/nfs/rpc_pipefs 2>/dev/null 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Note: here above, not remounting to /sysroot is probably NOT correct when having a NFS root; this hack is just to show what fixes the issue on regulard systems actually. Version-Release number of selected component (if applicable): dracut-network-049-202.git20220511.el8_6.x86_64 and before How reproducible: Always Steps to Reproduce: 1. Install dracut-config-generic package yum -y install dracut-config-generic 2. Rebuild the initramfs dracut -f 3. Setup persistent journal and boot twice mkdir -p /var/log/journal reboot ... reboot 4. Check ordering cycles at shutdown journalctl -b -1 | grep local-fs.target Actual results: -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found ordering cycle on var-lib-nfs-rpc_pipefs.mount/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on systemd-tmpfiles-setup.service/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Found dependency on local-fs.target/stop Sep 26 13:44:54 vm-rhel8 systemd[1]: local-fs.target: Job var-lib-nfs-rpc_pipefs.mount/stop deleted to break ordering cycle starting with local-fs.target/stop -------- 8< ---------------- 8< ---------------- 8< ---------------- 8< -------- Expected results: No ordering cycle

Release Notes

No. of Comments

Resolution

Won't Do