Loading...
Loading...
Cluster getting unusually fuller than expected. Enormous amount of data in a single SYSTEM snapshot. Automatic snapshot deletion is not running as the huge SYSTEM snapshot stays active (NOT in deleting state - being the largest system snap ID) Too many backlog entries in async delete queue. To Confirm the issue Check the entries count in async delete defer queue: # isi_for_array -s sysctl efs.util.defer_queue.dfq_async_delete.entries Check the Deferred File Queue (DFQ) sync delete thread limit (Expected Default: 64): # isi_for_array -s sysctl efs.util.defer_queue.dfq_async_delete.thread_limit Example below showing only one System Snapshot is being deleted by SnapshotDelete: isilon-71# isi job jobs view 3002 ID: 3002 Type: SnapshotDelete State: Running Impact: Low Policy: LOW Pri: 5 Phase: 2/2 Start Time: 2026-08-12T12:31:39 Running Time: 1d 3h 59m Participants: 1, 2, 3, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21 Progress: Deleted 0/0 LINs and 0/1 snapshot; 0 errors <<<<<< Single snapshot Waiting on job ID: - Description: Human Desc: isilon-71#isilon-71# isi job reports view 3002 SnapshotDelete[3002] phase 1 (2026-08-12T12:31:40) -------------------------------------------------- Snapshots processed 1 <<<<<< Single Snapshot LINs Total 0 Total bytes 7737799727153152 bytes (6.87P) Maximum LIN 12130451471 Minimum LIN 7731419080 Highest System Snapshot Snapid 63 <<<<<< Snapshot ID JE/Error Count 0 JE/Time elapsed 1 second JE/Time working 1 secondSnapshotDelete[3002] Job Summary -------------------------------- Final Job State Running Phase Executed 1
The issue is usually caused by steps taken in the past: - The DFQ sync delete thread limit was reduced from the default of 64. - The deletion of numerous large files at a rate that the reduced DFQ threads become unable to handle.
To avoid experiencing this issue: Never change the DFQ sync delete thread limit without approval from Dell Technical Support. - Too many threads could lead to overloading on some nodes. - Too few threads could lead to a large backlog in deferred queue resulting in a huge System snapshot. Workaround: If the cluster is already experiencing the issue: (1) Contact Dell Technical Support to evaluate whether the DFQ sync delete thread limit can be changed back safely. (2) Create a temporary snapshot schedule for an empty folder, such as /ifs/data/Isilon_Support/empty/, to create a new snapshot every hour and expire them in 2 hours. Rationale: - The creation of new snapshot would also create a new System Snapshot. - After each new System Snapshot is created, the deletion of large files would Copy on Write (CoW) to the new System Snapshot. - This will help to balance the data of deleted large files across multiple different System Snapshots. - This will help avoid bottlenecks to the Snapshot Tracking File (STF) Delta Block. - The older System snapshots will go into a deleted state which would also trigger SnapshotDelete jobs every hour when the non-system snaps expire. Permanent Solution: Dell Engineering is currently working on a permanent solution.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.