Loading...
Loading...
Avamar - Isilon - NDMP Incremental Backups Completed with Exceptions. Avamar backups are missing files during incremental backups. This condition could cause data to not be backed up and not be available for restore. During NDMP incremental backup with Isilon OneFs, the backup completed with exceptions or failed with the following Avtar error: avtar error: 2015-03-12 03:02:53 avtar Error <9982>: While processing _file_name_X or directory_name_X (inode XXXX713107), NDMP file history messages indicate the presence of a directory entry (inode:XXXX870544 name: Y) that doesn't appear in the data stream or in our previous backup of this directory. Severity Rating: High Impact Description: Avamar backups are missing files during incremental backups due to Isilon OneFs not detecting file case changes and not sending data stream for files or directories with case changes.
Root Cause of the Issue The root cause of the issue is related to how file system scanning is performed by the NAS for changes, and not the NDMP client. It is up to the NAS to determine what has changed, and the NDMP client only receives what the NAS tells it has changed. Avamar is sensitive during the incremental process and tries to find the previous data stream from the "file_name_X or directory_name_X" but instead finds a file or directory change case "File_Name_x or directory_name_x" and results in no valid backups. This condition is caused because Isilon OneFs is not detecting file case changes and not sending a data stream for files or directories with case changes. Specific Error Messages The issue is associated with the following Avtar error: 2015-03-12 03:02:53 avtar Error <9982>: While processing _file_name_X or directory_name_X (inode XXXX713107), NDMP file history messages indicate the presence of a directory entry (inode:XXXX870544 name: Y) that doesn't appear in the data stream or in our previous backup of this directory. 📋
Permanent Fix and Workarounds The final resolution is underway by both teams Avamar/Isilon to resolve this from the root cause. Permanent fix In Isilon OneFS v7.2.0.3 as listed in "Isilon OneFS Version 7.2.0.0 - 7.2.0.5 Release Notes" ID 147606. In Isilon OneFS v7.1.1.5 as listed in "Isilon OneFS Version 7.1.1.0 - 7.1.1.8 Release Notes" ID 131843. Workaround Options Option 1: Perform Level 0 backups Option 2: Customers to keep track of incremental backups that completed with exceptions Option 1. Perform Level 0 backups. Some errors or exceptions must be tracked by customer to maintain data integrity. To avoid performance issues split the Isilon NDMP backup into multiple target directories. Directory "/ifs" should not be a target. If it is, only a single stream will be used to backup up all data. Option 2: Track Incremental Backups Examine the logs to find the renamed files, then touch the files so that they appear changed in the next incremental backup. Do not allow more than 30 consecutive backups to "complete with exceptions", otherwise additional manual intervention is required for the software to find a valid base for the next incremental. As longer it is since the last valid "successful" backup, the more data that is sent by NAS during the next incremental. CAUTION: Be aware of the potential performance issues and data integrity when using these workarounds.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.