Issue

What were you trying to do that didn't work? This issue was found for ppc64le platform after an upgrade from rhel7.9(RHEL-7.9-updates-20240116.0 Server) to rhel8.10(RHEL-8.10.0-20240207.29). The test machine was borrowed from beaker, and I was using this script[1] to process the upgrade. [1] https://gitlab.cee.redhat.com/container-runtime-qe/leapp-rhel-upgrades/-/blob/main/7to8.sh?ref_type=heads I didn't find this issue on the other platforms, and it only happens after upgrade. If I directly run this on a rhel8.10(RHEL-8.10.0-20240207.29) ppc64le server without an upgrade, it can't be seen. Please provide the package NVR for which bug is seen: Before upgrade # rpm -q podman runc buildah skopeo systemd kernel containers-common podman-1.6.4-36.el7_9.ppc64le runc-1.0.0-70.rc10.el7_9.ppc64le buildah-1.11.6-12.el7_9.ppc64le skopeo-0.1.40-12.el7_9.ppc64le systemd-219-78.el7_9.9.ppc64le kernel-3.10.0-1160.108.1.el7.ppc64le containers-common-0.1.40-12.el7_9.ppc64le After upgrade # rpm -q podman runc buildah skopeo systemd kernel containers-common podman-4.9.2-1.module+el8.10.0+21261+843dd79b.ppc64le runc-1.1.12-1.module+el8.10.0+21251+62b7388c.ppc64le package buildah is not installed package skopeo is not installed systemd-239-79.el8.ppc64le kernel-4.18.0-539.el8.ppc64le containers-common-1-58.module+el8.10.0+21077+98b84d8a.ppc64le How reproducible: always Steps to reproduce `podman run -dt --name testubi ubi8 echo foo` Expected results the container can be successfully run Actual results # podman run --log-level debug -dt --name testubi ubi8 echo foo INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug -dt --name testubi ubi8 echo foo) DEBU[0000] Using conmon: "/usr/bin/conmon" INFO[0000] Using boltdb as database backend DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is not being used INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2024-02-12 08:44:16.891760216 -0500 EST [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] [] false false false [] map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Pulling image ubi8 (policy: missing) DEBU[0000] Looking up image "ubi8" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Looking up image "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "registry.access.redhat.com/ubi8:latest" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "registry.access.redhat.com/ubi8:latest" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Looking up image "ubi8" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] using systemd mode: false DEBU[0000] setting container name testubi DEBU[0000] No hostname set; container's hostname will default to runtime default DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" DEBU[0000] Allocated lock 3 for container 569b155bf67a54280b39f74a9fb6921aef8db17c596bcec72e869f386bb326cd DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported DEBU[0000] Check for idmapped mounts support DEBU[0000] Failed to create container testubi(569b155bf67a54280b39f74a9fb6921aef8db17c596bcec72e869f386bb326cd): the container name "testubi" is already in use by 57ac3b8854f4242360a17dacf07a00cef9d690fb1b8cd3dbd86485025e0e19ee. You have to remove that container to be able to reuse that name: that name is already in use Error: creating container storage: the container name "testubi" is already in use by 57ac3b8854f4242360a17dacf07a00cef9d690fb1b8cd3dbd86485025e0e19ee. You have to remove that container to be able to reuse that name: that name is already in use, or use --replace to instruct Podman to do so. DEBU[0000] Shutting down engines You have mail in /var/spool/mail/root [root@ibm-p8-pvm-09-guest-14 ~]# podman rm -af 7a0708e033a8f83a78e52028051cc57d47e6c30ddcb05147dfc75f8d65c390c6 57ac3b8854f4242360a17dacf07a00cef9d690fb1b8cd3dbd86485025e0e19ee 27d6ff8942d8ce2e0f00e6fc67c15e773aa5f55efa4cf70b2bfb9abaf9bc1036 [root@ibm-p8-pvm-09-guest-14 ~]# podman run --log-level debug -dt --name testubi ubi8 echo foo INFO[0000] podman filtering at log level debug DEBU[0000] Called run.PersistentPreRunE(podman run --log-level debug -dt --name testubi ubi8 echo foo) DEBU[0000] Using conmon: "/usr/bin/conmon" INFO[0000] Using boltdb as database backend DEBU[0000] Initializing boltdb state at /var/lib/containers/storage/libpod/bolt_state.db DEBU[0000] Overriding run root "/run/containers/storage" with "/var/run/containers/storage" from database DEBU[0000] Overriding tmp dir "/run/libpod" with "/var/run/libpod" from database DEBU[0000] Using graph driver overlay DEBU[0000] Using graph root /var/lib/containers/storage DEBU[0000] Using run root /var/run/containers/storage DEBU[0000] Using static dir /var/lib/containers/storage/libpod DEBU[0000] Using tmp dir /var/run/libpod DEBU[0000] Using volume path /var/lib/containers/storage/volumes DEBU[0000] Using transient store: false DEBU[0000] [graphdriver] trying provided driver "overlay" DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that overlay is supported DEBU[0000] Cached value indicated that metacopy is not being used DEBU[0000] Cached value indicated that native-diff is not being used INFO[0000] Not using native diff for overlay, this may cause degraded performance for building images: failed to mount overlay: invalid argument DEBU[0000] backingFs=xfs, projectQuotaSupported=false, useNativeDiff=false, usingMetacopy=false DEBU[0000] Initializing event backend file DEBU[0000] Configured OCI runtime crun initialization failed: no valid executable found for OCI runtime crun: invalid argument DEBU[0000] Configured OCI runtime kata initialization failed: no valid executable found for OCI runtime kata: invalid argument DEBU[0000] Configured OCI runtime runsc initialization failed: no valid executable found for OCI runtime runsc: invalid argument DEBU[0000] Configured OCI runtime youki initialization failed: no valid executable found for OCI runtime youki: invalid argument DEBU[0000] Configured OCI runtime krun initialization failed: no valid executable found for OCI runtime krun: invalid argument DEBU[0000] Configured OCI runtime crun-wasm initialization failed: no valid executable found for OCI runtime crun-wasm: invalid argument DEBU[0000] Configured OCI runtime runj initialization failed: no valid executable found for OCI runtime runj: invalid argument DEBU[0000] Configured OCI runtime ocijail initialization failed: no valid executable found for OCI runtime ocijail: invalid argument DEBU[0000] Using OCI runtime "/usr/bin/runc" INFO[0000] Setting parallel job count to 25 DEBU[0000] Successfully loaded network podman: &{podman 2f259bab93aaaaa2542ba43ef33eb990d0999ee1b9924b557b7be53c0b7a1bb9 bridge cni-podman0 2024-02-12 08:44:16.891760216 -0500 EST [{{{10.88.0.0 ffff0000}} 10.88.0.1 <nil>}] [] false false false [] map[] map[] map[driver:host-local]} DEBU[0000] Successfully loaded 1 networks DEBU[0000] Pulling image ubi8 (policy: missing) DEBU[0000] Looking up image "ubi8" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Loading registries configuration "/etc/containers/registries.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/000-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/001-rhel-shortnames.conf" DEBU[0000] Loading registries configuration "/etc/containers/registries.conf.d/002-rhel-shortnames-overrides.conf" DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Looking up image "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "registry.access.redhat.com/ubi8:latest" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "registry.access.redhat.com/ubi8:latest" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Looking up image "ubi8" in local containers storage DEBU[0000] Normalized platform linux/ppc64le to {ppc64le linux [] } DEBU[0000] Trying "registry.access.redhat.com/ubi8:latest" ... DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage DEBU[0000] Found image "ubi8" as "registry.access.redhat.com/ubi8:latest" in local containers storage ([overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2) DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] Inspecting image 72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2 DEBU[0000] using systemd mode: false DEBU[0000] setting container name testubi DEBU[0000] No hostname set; container's hostname will default to runtime default DEBU[0000] Loading seccomp profile from "/usr/share/containers/seccomp.json" DEBU[0000] Allocated lock 0 for container 55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc DEBU[0000] parsed reference into "[overlay@/var/lib/containers/storage+/var/run/containers/storage:overlay.mountopt=nodev,metacopy=on]@72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] exporting opaque data as blob "sha256:72ceb5804a843f408a388a4037df87b8566985a9eefc8b8d8bd23226ab2b1df2" DEBU[0000] Cached value indicated that idmapped mounts for overlay are not supported DEBU[0000] Check for idmapped mounts support DEBU[0000] Created container "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" DEBU[0000] Container "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" has work directory "/var/lib/containers/storage/overlay-containers/55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc/userdata" DEBU[0000] Container "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" has run directory "/var/run/containers/storage/overlay-containers/55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc/userdata" DEBU[0000] Made network namespace at /run/netns/netns-5d82d4c6-3431-a321-4d48-8deaecf4bcf1 for container 55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc WARN[0000] Ignoring global metacopy option, not supported with booted kernel 3.10.0-1160.108.1.el7.ppc64le #1 SMP Thu Jan 4 14:59:31 EST 2024 DEBU[0000] overlay: mount_data=lowerdir=/var/lib/containers/storage/overlay/l/GQDXWVZXUFOFP7IDVYXRO4WBSG,upperdir=/var/lib/containers/storage/overlay/6ddcd5486c02920e3731d452049805c3500a0e5edd8341782c90f7404439ddab/diff,workdir=/var/lib/containers/storage/overlay/6ddcd5486c02920e3731d452049805c3500a0e5edd8341782c90f7404439ddab/work,nodev,context="system_u:object_r:container_file_t:s0:c22,c562" DEBU[0000] Mounted container "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" at "/var/lib/containers/storage/overlay/6ddcd5486c02920e3731d452049805c3500a0e5edd8341782c90f7404439ddab/merged" DEBU[0000] Created root filesystem for container 55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc at /var/lib/containers/storage/overlay/6ddcd5486c02920e3731d452049805c3500a0e5edd8341782c90f7404439ddab/merged WARN[0000] Failed to load cached network config: network podman not found in CNI cache, falling back to loading network podman from disk DEBU[0001] Unmounted container "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" DEBU[0001] Network is already cleaned up, skipping... DEBU[0001] Cleaning up container 55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc DEBU[0001] Network is already cleaned up, skipping... DEBU[0001] Container 55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc storage is already unmounted, skipping... DEBU[0001] ExitCode msg: "plugin type=\"bridge\" failed (add): cni plugin bridge failed: running [/usr/sbin/iptables -t nat -a postrouting -s 10.88.0.4 -j cni-a818395048b2640a31d2421e -m comment --comment name: \"podman\" id: \"55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc\" --wait]: exit status 4: iptables v1.8.5 (nf_tables): rule_append failed (invalid argument): rule in chain postrouting\n" Error: plugin type="bridge" failed (add): cni plugin bridge failed: running [/usr/sbin/iptables -t nat -A POSTROUTING -s 10.88.0.4 -j CNI-a818395048b2640a31d2421e -m comment --comment name: "podman" id: "55f2fb1296140ead23a79f5a40f09ea328cd4e0d050e33dcba50a2e16488effc" --wait]: exit status 4: iptables v1.8.5 (nf_tables): RULE_APPEND failed (Invalid argument): rule in chain POSTROUTING DEBU[0001] Shutting down engines

Release Notes

No. of Comments

Resolution

Won't Do