Loading...
Loading...
Deduplication job keeps failing with the following error: Dedupe failing "[MAJOR] dedupe CreateDedupeRequestForDigest/1846 failed to check index status Handle must be released and re-acquired" The job might succeed in subsequent runs. Recent finished jobs: ID Type State Time ----------------------------------------------------- 4136 Dedupe Failed 2019-01-21T14:59:06 4137 Dedupe Failed 2019-01-21T15:02:51 4138 Dedupe Failed 2019-01-21T15:05:38 4139 Dedupe Failed 2019-01-21T15:30:41 4140 Dedupe Failed 2019-01-21T15:36:10 Errors seen in /var/log/messages: 2019-01-21T15:35:52+01:00 <3.3> lab-5 isi_job_d[64403]: [MAJOR] dedupe CreateDedupeRequestForDigest/1846 failed to check index status Handle must be released and re-acquired 2019-01-21T15:35:52+01:00 <3.3> lab-5 isi_job_d[64403]: dedupe_item_process:3734: BEDRSCommonalityTaskItemProcessing failed with status 22 2019-01-21T15:35:55+01:00 <3.5> lab-5 isi_job_d[64382]: Coord (coord_from_director_abort_handler, 3237): Received an abort message from devid 4 for jid 4140: 19-01-21 15:35:55 Dedupe[4140] Node 4 (4) task 5-56: Invalid argument from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from dedupe_item_process (/b/mnt/src/isilon/bin/isi_job_d/dedupe_common.c:3744) from worker_process_task_item (/b/mnt/src/isilon/bin/isi_job_d/worker.c:985) 2019-01-21T15:35:55+01:00 <3.3> lab-5 isi_job_d[64382]: Coord (coord_fail_job, 1509): Dedupe[4140] Failing job: 19-01-21 15:35:55 Dedupe[4140] Node 4 (4) task 5-56: Invalid argument from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from dedupe_item_process (/b/mnt/src/isilon/bin/isi_job_d/dedupe_common.c:3744) from worker_process_task_item (/b/mnt/src/isilon/bin/isi_job_d/worker.c:985) 2019-01-21T15:35:55+01:00 <3.5> lab-5 isi_job_d[64382]: Coord (coord_from_director_abort_handler, 3237): Received an abort message from devid 3 for jid 4140: 19-01-21 15:35:55 Dedupe[4140] Node 3 (3) task 5-45: Invalid argument from bam_revalidate_ivec (/b/mnt/src/sys/ifs/bam/bam_vget.c:1662): ENOENT from bam_revalidate_inode (/b/mnt/src/sys/ifs/bam/bam_vget.c:2421): ENOENT from bam_update_inode_hint (/b/mnt/src/sys/ifs/bam/bam_vget.c:1481): ENOENT from bam_vget_valid_pref_hint (/b/mnt/src/sys/ifs/bam/bam_vget.c:947): ENOENT from pctl2_lin_open_hint (/b/mnt/src/sys/ifs/bam/bam_pctl.c:4362): ENOENT from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/sr...snipped...2019-01-21T15:36:09+01:00 <3.3> lab-5 isi_job_d[64382]: Coord (coord_fail_job, 1509): Dedupe[4140] Failing job: 19-01-21 15:36:08 Dedupe[4140] Node 1 (1) task 5-107: Invalid argument from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from coalmem_book (/b/mnt/src/sys/ifs/bam/bam_coalesce.c:1901): EAGAIN from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from ifm_leaf_get_entry (/b/mnt/src/sys/ifs/ifm/ifm_metatree.c:1169): ENOENT from dedupe_item_process (/b/mnt/src/isilon/bin/isi_job_d/dedupe_common.c:3744) from worker_process_task_item (/b/mnt/src/isilon/bin/isi_job_d/worker.c:985) 2019-01-21T15:36:10+01:00 <24.5> lab-5 isi_job_d[64382]: Dedupe[4140] Failed
The deduplication job fails to index the cluster.
Workaround: These steps reset the dedupe_metadata and force a rebuild. This workaround does not negatively impact the cluster, clients, or previously deduplicated data. Cancel any running deduplication job: # isi job cancel dedupe Make a backup copy of the existing dedupe_metadata . * Make a backup directory: # mkdir /ifs/data/Isilon_Support/<SR#> (where <SR#> is the Support case #) Copy the existing metadata over: (This may be large and may take several minutes to copy) # cp -R /ifs/.ifsvar/modules/bedrs/dedupe_metadata /ifs/data/Isilon_Support/<SR#> Remove the original copy of the dedupe_metadata . Watch for typo errors. (This may take a few minutes for the command to complete.) # rm -rf /ifs/.ifsvar/modules/bedrs/dedupe_metadata/* Schedule a new deduplication job. # isi job start dedupe Monitor the job using the following command. Deduplication automatically rebuilds the metadata so the first couple iterations of the job may be prolonged and not show significant progress. # isi job jobs view dedupe * The backup copy of dedupe_metadata can be between 1~9T in size and consume disk space. Once the deduplication issue is resolved, it is safe to delete the backup copy from /ifs/data/Isilon_Support/<SR#> and reclaim the disk space.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.