Symptoms

Isilon/PowerScale node panics with the following stack: panic @ time 1636820409.559, thread 0xfffff80b615e8000: Fatal trap 12: page fault while in kernel mode time = 1636820409 cpuid = 3, TSC = 0x1825289de5b108 Panic occurred in module kernel loaded at 0xffffffff80200000: Stack: -------------------------------------------------- kernel:trap_fatal+0xa3 kernel:trap_pfault+0x48 kernel:trap+0x314 kernel:curb_cache_free_nondelta+0x43 kernel:bam_update_inode_lock_loss_cb+0x161 kernel:lin_lock_deltaread_lock_loss_probe+0x120 kernel:lki_decide_llcb_type_and_probe+0x26c kernel:lki_lock_loss_add_waiters+0x4b kernel:lki_unlock_idds_entry+0x258 kernel:lki_shrink_lazy_queue+0x439 kernel:_lki_unlock_owner+0x413 kernel:lki_unlock_impl+0x56b kernel:lk_unlock+0xbf kernel:lin_lock_release_locker+0x28 kernel:ifs_vnop_wrapunlock+0xfe kernel:VOP_UNLOCK_APV+0x97 kernel:vputx+0x154 kernel:bam_vput+0x20 kernel:putfd+0x17 kernel:_pctl2_start_restripe+0x35a kernel:_sys_pctl2_start_restripe+0x58a kernel:amd OR panic @ time 1640560360.736, thread 0xfffff805f25cd000: Assertion Failure time = 1640560360 cpuid = 5, TSC = 0x5afba0fd75a32d Panic occurred in module kernel loaded at 0xffffffff80200000: Stack: -------------------------------------------------- kernel:isi_assert_halt+0x2e kernel:curb_cache_lin_llcb+0x56 kernel:bam_update_inode_lock_loss_cb+0x161 kernel:lin_lock_deltaread_lock_loss_probe+0x120 kernel:lki_decide_llcb_type_and_probe+0x26c kernel:lki_lock_loss_add_waiters+0x4b kernel:lki_unlock_idds_entry+0x258 kernel:lki_shrink_lazy_queue+0x439 kernel:_lki_unlock_owner+0x413 kernel:lki_unlock_impl+0x56b kernel:lk_unlock+0xbf kernel:lin_lock_release_locker+0x28 kernel:ifs_vnop_wrapunlock+0xfe kernel:VOP_UNLOCK_APV+0x97 kernel:vputx+0x154 kernel:bam_vput+0x20 kernel:putfd+0x17 kernel:_sys_ifs_sbt_get_entry+0x315 kernel:amd64_syscall+0x9f2

Cause

Caused by a known issue PSCALE-62304, resolved in 9.1.0.13 (November patch) and also in 9.2.0.0(resolved in 9.1.0.13 PATCH: [9.1.0.13_GA-RUP_2021-11][Multiple User space and Kernel Fixes](November 2021) The unexpected panic reboot is caused by a race condition while releasing a cache from an internal-use cluster statistics inode.This issue in code has been fixed in the following versions or later: OneFS 9.1.0.13 via PSCALE-62304OneFS 9.2 via PSCALE-49586OneFS 9.3 via PSCALE-50848

Resolution

Advise customer to upgrade to one of the releases above.There are no workarounds available.

Support Cases