Loading...
Loading...
During an install operation, "EXT4-fs error" and "Buffer I/O error" are seen and the install operation fails with a 'Structure needs cleaning' error: Examples: [126400.843219] EXT4-fs error (device dm-12): ext4_lookup:1609: inode #401229: comm instorchagt: deleted inode referenced: 414190 [126405.192299] Buffer I/O error on device dm-12, logical block 602496 instorch[277]: %INFRA-INSTALL-3-ACTION_FAILED : Packaging operation 27.1.1 failed - Internal error on 1 node. First error: 0/RP0/CPU0: [Errno 117] Structure needs cleaning:
This issue may occur on XR7 platforms running releases before 7.1.2 where this issue is fixed. The issue can only occur if the filesystem has been exhausted of memory. The issue is seen when the 'install-rootfs-thinpool' is full. This can be identified by running the lvs command from the shell, or via 'run lvs' from the XR prompt, an example is in the further problem description. The particular filesystem exhaustion tracked by this issue can only occur if the system has had several aborted 'apply by reload' install events over its uptime. Specifically: - These aborts must have occurred after the reloaded system reached running state with new software - These aborts must cause the device to fail back to the old running software before the new software has been committed via install commit. In this case, a stale system snapshot is left in the 'install-rootfs-thinpool' and needs to be purged.
There is no true workaround. The user can periodically monitor the amount of disk used by the 'install-rootfs-thinpool' using the lvs command to and cross-reference it with the boot menu to check for stale snapshots. Stale snapshots can be purged by using the unix lvremove command. An example is included in the further problem description.
At the start of an install operation, a snapshot of the system is taken to allow either a user-initiated rollback or an emergency fallback to occur. Snapshots are cleared when the user commits to the new software (install commit), or aborts their changes. If an install operation causes a reload and that reload succeeds, but brings the system up in a bad state (e.g. a bad bugfix causes a process to abnormally terminate multiple times), the system will automatically fallback to the fallback snapshot. If this occurs before a user initiated abort or commit to new software, a new fallback snapshot may be created without deleting the original, effectively leaking a snapshot. The situation on any device can be monitored actively and stale snapshots can be purged to mitigate this issue Outside of an install operation, the only snapshots should be those referenced in the boot rootfs names file: /boot/install-boot-rootfs-names. If a user suspects they have stale snapshots, cross-referencing that file with the output of lvs can show stale snapshots. In the example below, the thinpool is at 79.82% capacity (so no issue yet) but we have two stale snapshots, install-rootfs-snapshot-1592489429 and install-rootfs-snapshot-1592064769 (so this issue is primed to occur if another stale snapshot is created). The valid snapshot is install-rootfs-snapshot-1594004796. The user can cross reference /boot/install-boot-rootfs-names and lvs: [node0_RP0_CPU0:~]$cat /boot/install-boot-rootfs-names latest: main-xr-vg/install-rootfs-snapshot-1594004796 fallback: main-xr-vg/install-rootfs-snapshot-1594004796 [node0_RP0_CPU0:~]$lvs LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert boot-lv main-xr-vg -wi-ao---- 956.00m install-data-disk1 main-xr-vg Vwi-aotz-- 9.21g install-data-thinpool 19.34 install-data-log main-xr-vg Vwi-aotz-- 1.86g install-data-thinpool 57.08 install-data-scratch main-xr-vg Vwi-aotz-- 956.00m install-data-thinpool 28.01 install-data-thinpool main-xr-vg twi-aotz-- 11.96g 25.98 12.70 install-rootfs-snapshot-1592064769 main-xr-vg Vwi---tz-k 14.90g install-rootfs-thinpool install-rootfs-snapshot-1592489429 main-xr-vg Vwi---tz-k 14.90g install-rootfs-thinpool install-rootfs-snapshot-1592064769 install-rootfs-snapshot-1594004796 main-xr-vg Vwi-aotz-k 14.90g install-rootfs-thinpool install-rootfs-snapshot-1592489429 28.84 install-rootfs-thinpool main-xr-vg twi-aotz-- 14.90g 79.82 42.41 Determine these stale snapshots, and purge them using lvremove: lvremove -f main-xr-vg/install-rootfs-snapshot-1592064769 lvremove -f main-xr-vg/install-rootfs-snapshot-1592489429 If the lvremove fails, passing a double force (lvremove -ff) will remove the file.
Cisco Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.