Symptoms

This article provides a concise overview of Data Domain Retention Lock (RL) functionality and answers to related frequently asked questions (FAQ).

Cause

This article provides a concise overview of Data Domain Retention Lock (RL) functionality and answers to related frequently asked questions (FAQ).

Resolution

What is a retention lock?Retention lock is a feature used on Data Domain Restorers (DDRs) to prevent modification or deletion of certain sets of files for a predetermined period. That is, retention locked files are read-only until their retention period expires.What are the different versions of retention lock?Retention lock is available for two different functions: Governance: The less strict of the two retention lock functions (that is locks against files can be reverted if necessary).Compliance: The stricter of the two functions which adhere to several common regulatory standards. That is, locks against files cannot be reverted. The DDR must be configured with a 'security officer' user who must authenticate certain commands. There are various restrictions on other functionality to prevent locked data from being removed or locks being reverted early. Note: Compliance mode is only available in DDOS 5.3 (and later).Each function of the retention lock requires a separate license key.Retention lock functionality is enabled on a per MTree basis.A single system can use both governance and compliance mode against separate MTrees. However, it must have separate governance and compliance licenses installed.Do not enable any kind of DD Retention Lock on MTrees used to store Avamar data unless called for in the Avamar documentation, as part of getting this feature running on Avamar. Enabling DD RL on any such MTree without following the right Avamar process can render the MTree unusable for backups, and incur the need for lengthy recovery. This is specially true if enabling Automatic Retention Lock (ARL) for an Avamar MTree.Retention lock functionality may not be supported against MTrees that are used to store data using older versions of Avamar or Protection Software data on an Integrated Data Protection Appliance or PowerProtect Data Protection Series appliance. This can prevent Avamar or Protection Software inside Integrated Data Protection Appliance from functioning as expected and go into READONLY state. Which data access protocols are supported with retention lock? The NFS, CIFS, and DD Boost protocols are fully supported against MTrees using retention lock governance or compliance mode.The Virtual Tape Library (VTL) protocol is only supported against MTrees using retention lock governance mode. Automatic retention lock is not supported on Data Domain VTL. See the Data Domain Administration Guide to determine how to unlock retention-locked tapes such that they can be written to. Retention lock compliance mode meets regulatory standards:The list of regulatory standards that retention lock compliance mode meets includes the following: SEC 17a-4(f)CFTC Rule 1.31bFDA 21 CFR Part 11Sarbanes-Oxley ActIRS 98025 and 97-22ISO Standard 15489-1MoREQ2010 For full details of certification information, contact your contracted support provider.How is retention lock governance enabled? A retention lock governance license is added to the DDR.Retention lock governance mode is enabled against any required MTree: # mtree retention-lock enable mode governance mtree [mtree] How is retention lock compliance enabled? There are specific instructions for some newer Data Domain models with iDRAC.A retention lock compliance license is added to the DDR.A user with the role of 'security' should be created (assuming such a user does not exist): (ADMIN USER) # user add [username] role security The user with the role of 'security' should log in to the DDR and enable security user authorization: (SECURITY USER): # authorization policy set security-officer enabled The system should be configured for retention lock compliance mode. Once the following command runs to completion, the system reboots automatically: (ADMIN USER) # system retention-lock compliance configure Once the system has rebooted, retention lock compliance mode should be enabled on the system: (ADMIN USER) # system retention-lock compliance enable Note: For newer DDOS releases, this task must be performed with the Data Domain UI. Administration > Compliance Retention lock compliance mode is enabled against any required MTree: (ADMIN USER) # mtree retention-lock enable mode compliance mtree [mtree] How is it possible to determine which MTrees have retention lock enabled?MTrees with retention lock enabled are indicated in the output of mtree list, for example: sysadmin@ddxxxx# mtree list Name Pre-Comp (GiB) Status Tenant-Unit --------------------------------- -------------- ------- ----------- ... /data/col1/rich-retention-lock 0.0 RW/RLGE - /data/col1/rl_test 0.0 RW/RLGD - /data/col1/rl_test_comp 0.0 RW/RLCE - /data/col1/test 3.1 RW/RLGE - ... --------------------------------- -------------- ------- ----------- ... RLGE : Retention-Lock Governance Enabled RLGD : Retention-Lock Governance Disabled RLCE : Retention-Lock Compliance Enabled Can an MTree with Retention Lock Governance enabled be converted to Retention Lock Compliance?As stated in the DDOS Administration Guide, this is not possible.Can an MTree with Retention Lock Compliance enabled be converted to Retention Lock Governance?As stated in the DDOS Administration Guide, this is not possible.How are file retention or lock periods set?Once a retention lock is enabled against an MTree, a minimum and maximum retention period must be set. These periods dictate the minimum and maximum time a file within the MTree can be locked for. For example: # mtree retention-lock set min-retention-period [period] mtree [mtree] # mtree retention-lock set max-retention-period [period] mtree [mtree] Periods can be given in various units as follows: 1 min1 hr1 day1 mo1 year Note: A minimum retention period cannot be less than 12 hours.A maximum retention period cannot be greater than 70 years.The minimum retention period must be less than the maximum retention period.Retention periods for each MTree are set in the same way regardless of the flavor of retention lock being used. How can existing retention periods be displayed?This can be done using the following two commands: # mtree retention-lock show min-retention-period mtree [mtree] # mtree retention-lock show max-retention-period mtree [mtree] For example: sysadmin@dd630# mtree retention-lock show min-retention-period mtree /data/col1/rl_test Retention-lock min-retention-period of mtree /data/col1/rl_test is: 720 minutes. sysadmin@dd630# mtree retention-lock show max-retention-period mtree /data/col1/rl_test Retention-lock max-retention-period of mtree /data/col1/rl_test is: 30 days. How are files within an MTree with retention lock enabled locked? When retention lock is enabled against an MTree, existing files within the MTree are not automatically locked (that is, all preexisting files remain read or write).When a new file is written to an MTree with retention lock enabled, the file is not automatically retention locked. That is, the new file remains as read or write. Note: Starting with DDOS 6.2.0.20, there is a feature in DDOS called "automatic retention lock," which may set a lock on all written files automatically. See the "Automatic Retention Lock" section further down in this KB article to learn more. To retention lock a specific file, the atime of the file must be modified to match the date, and time the file should be retention locked. That is the date and time until which the file should remain read-only. Until the time is modified in this way, the file cannot be retention locked (and can be modified or removed). A file's atime can be changed from an NFS or CIFS client using the 'touch' command: # touch -a -t [expiry time] [file to be locked] For example, to set atime of /data/col1/rl_test/testfile to 07:05 on 8 June: # touch -a -t 06080705 /data/col1/rl_test/testfile The period from current time to future atime must be within the minimum and maximum retention periods for the MTree. Otherwise, an error is generated when modifying the files atime: # touch -a -t 08080705 /data/col1/rl_test/testfile touch: setting times of `/data/col1/rl_test/testfile': Permission denied A corresponding message is also shown in the Data Domain File System (DDFS) log files: 06/07 13:44:57.197 (tid 0x2b28ee5258c0): Attempt to set atime of /data/col1/rl_test/testfile to larger than maximum retention period of mtree. CIFS (Windows) clients do not, by default, include a touch command or utility however several such utilities are freely available for download from various third-party websites. Note: Files cannot be retention locked until they are written to the DDR. It is not possible to create an empty file, retention lock that file, then write data to the file. Which backup applications support automatically retention locking files after writing them to a DDR?Data Domain Retention Lock is compatible with industry-standard, NAS-based Write-Once-Read-Many (WORM) protocols. Integration is qualified with archive applications such as Symantec Enterprise Vault, SourceOne, Cloud Tiering Appliance, or DiskXtender.For Dell NetWorker, both governance and compliance modes are supported.As of June 2024, recent Avamar versions support both Data Domain Retention Lock Compliance and Governance) as part of Avamar "Immutable Backups" feature. See the article below and the Avamar documentation for details:Avamar and Data Domain: Enabling Avamar Immutable Backups and Data Domain Compliance Mode Retention LockIt is critical that the steps to enable the Avamar "Immutable Backups" feature are followed strictly. Failing to do so may end up with an Avamar MTree in the DD which cannot be further written to, or which has operational issues which force lengthy recovery. Avamar does not work with DD Automatic Retention Lock (ARL), and ARL must not be enabled for any Avamar MTree on a DD.Customers using other backup applications which do not natively support Data Domain retention lock can also develop custom scripts to use Data Domain Retention Lock to manually set retention periods of files. In this scenario, ensure that custom scripts set file's atime such that they are unlocked before the backup application attempting to delete the file. Failure to do this can result in the backup application attempting to delete locked files (which fails); The file is left on the DDR indefinitely consuming disk space. See Data Domain administration guide.Automatic Retention LockFor backup applications which do not natively support the Data Domain retention lock feature, it has always been an issue for customers to leverage the feature. The backup administrator must configure scripts to set the retention lock on newly ingested files for an MTree. The lock must be to be set so that it expires shortly before the backup is to be expired (and deleted) by the backup administrator.ARL does set a lock on each and every file written to the MTree after the feature has been enabled, preventing the file to be changed or removed for a given period of time after the cooling-off period configured has passed. That means ARL must not be enabled for workloads or backup applications that do must update some files repeatedly over time, for example, on VTL pools (VTL tape files are repeatedly written to), or for backup applications which keep metadata information alongside customer backup files themselves (such as NetWorker or Avamar). Enabling ARL in these instances to have important files locked for some time, and when these files must be written to later for other backups, the write fails, and so will any subsequent backups.To help backup administrators lower the burden of keeping retention lock son backup files and bring the DDOS up on feature parity with other vendors, since DDOS 6.2.0.20 there is a feature that can be enabled from the CLI for each MTree with Retention Lock configured, to set a lock on every file ingested (for a given period), after some predetermined amount of time has passed since the file write to disk was completed. This way, administrators do not have to worry anymore about manually (or scripted) setting the retention lock, and this can occur automatically without the cooperation of the backup application. Before DDOS 7.8, Automatic Retention Lock cannot be used on DD Boost Logical Storage Units (LSU) and attempting to enable this returns an error stating that it is not supported.From 7.8 onwards, ARL is supported for DD Boost LSUs.(ARL used on Target DDs for Managed File Replication (MFR), like NW clone should have a long enough "automatic-lock-delay" to ensure that clone operations are complete for the backup set before lock is set on the files. Example: A small file that is part of a backup set finishes replicating quickly while a larger file takes longer, then the first file is retention locked by the time the larger file finishes replicating and encounters an error when NW attempts to move all the files of the backup set to the final archival directory.)Automatic retention lock is not supported on Data Domain VTL.On applicable versions, there are additional options in the mtree retention-lock CLI, as shown below. This feature can be configured through the UI as well by choosing "Automatic" instead of "Manual" in the "Use" option: # mtree retention-lock set {min-retention-period | max-retention-period | automatic-retention-period | automatic-lock-delay} mtree Automatic retention lock feature locks a file immediately after a preconfigured cool-off period expires (automatic-lock-delay). After the file is written to a retention lock enabled MTree, and the lock is valid for "automatic-retention-period" from the moment it was set, if the value is within the "min-retention-period" and "max-retention-period" values set for the MTree, the lock occurs.For more usage and general details about the feature, check the corresponding DDOS Administration Guide. This feature is not well suited for situations in which the same MTree is used as the target for backups which should either have lock sets for different periods, or backups which should and backups which should have not a lock set to start with.What can or cannot be done against a locked file? Retention-locked files are read-only and cannot be modified or deleted.Once a file's retention period expires, it is 'unlocked' - when in an unlocked state the file still cannot be modified however it can be deleted. The DDR does not automatically delete a file when its retention period expires (subsequent deletion has to be performed from a client system or backup application).Once set the retention period for a specific file cannot be reduced (that is the files atime cannot be brought forwards).Retention periods can, however, be increased (atime can be increased up to the maximum retention period for the MTree).Ownership and permission settings for a file can continue to be modified while the file is lockedIt is only possible to rename or delete a directory in a retention lock enabled MTree if that directory contains no files. If the directory contains files (even if these files are not retention locked) the rename, or delete of the directory failsEven if it does not change the file contents itself, it is not allowed to change the name (rename) files which have a lock set, but the rename is also disallowed once the file's lock has expired. For no-longer locked files, the only operation which is allowed is a delete. This is changed in DDOS 7.7.4 when, for no-longer locked files, a rename of the file is permitted. Is it possible to completely remove the retention lock against a file or set of files?It is possible to 'revert' (remove) the retention lock against files in MTrees using governance mode - this is done with the following command: # mtree retention-lock revert [path] Once the retention lock against a file is removed, it can be modified or deleted as normal. If this command is run against a directory, then all files within that directory and all subdirectories have their retention locks removed.It is not possible to revert a retention lock against files in MTrees using compliance mode - if this is attempted a corresponding error is displayed: # mtree retention-lock revert /data/col1/rl_test_comp/testfile This operation is not allowed. Mtree is in retention-lock compliance mode. What happens if a retention-locked file is attempted to be modified or removed?Any attempt to modify or delete a retention-locked file causes a corresponding permission denied error: # echo " test2" >> /data/col1/rl_test/testfile bash: testfile: Permission denied # rm testfile rm: remove write-protected regular file `testfile'? y rm: cannot remove `testfile': Permission denied DDFS logs indicate that the operation has failed due to the file being retention locked: 06/07 07:06:59.756 (tid 0x2b5a77605d50): Atime of retention-lock file /data/col1/rl_test/testfile is not expired. 06/07 07:07:42.504 (tid 0x2b5a79111390): Atime of retention-lock file /data/col1/rl_test/testfile is not expired. Is it possible to list all files which are retention locked?Yes, this can be performed using the mtree retention-lock report generate retention-details command: mtree retention-lock report generate retention-details mtrees { | all} [format {text | tsv | csv}] [output-file ] Report detailed information of retention-lock files. For example to list details of all locked files in the /data/col1/rl_test MTree the following would be performed: sysadmin@ddxxxx# mtree retention-lock report generate retention-details mtrees /data/col1/jftest Report generated on: Fri Jul 1 14:19:31 2016 Report for mtree: /data/col1/jftest File Path Mode Size(Bytes) Expiration Date /data/col1/jftest/file1 governance 10521456 Sat Jul 2 22:35:48 2016 /data/col1/jftest/testdir/file2 governance 10521680 Sat Jul 2 22:35:42 2016 /data/col1/jftest/file3 governance 10521820 Sun Jul 10 22:36:09 2016 Total files: 3 Is it possible to completely disable retention lock for an MTree (after it is enabled)?Yes, for MTrees using governance mode, this is performed using the mtree retention-lock disable command: # mtree retention-lock disable mtree [mtree] For example: sysadmin@xxxx# mtree retention-lock disable mtree /data/col1/rl_test Retention-lock feature is disabled (previously enabled) for mtree /data/col1/rl_test. Once disabled, MTree list indicates that retention lock was used against the MTree but has since been disabled, that is: sysadmin@ddxxxx# mtree list Name Pre-Comp (GiB) Status Tenant-Unit --------------------------------- -------------- ------- ----------- ... /data/col1/rl_test 0.0 RW/RLGD - ... Note: Once the retention lock is disabled against an MTree: New files written to the MTree cannot be retention locked.Files which are already locked stay locked for their previously defined retention period (that is all locks are not automatically reverted when retention lock is disabled against an MTree).Existing locks against files in an MTree where retention lock is disabled cannot be reverted. All necessary reversions must be done before the retention lock being disabled: sysadmin@ddxxxx# mtree retention-lock revert /data/col1/rl_test/testfile **** Retention-lock feature is disabled (previously enabled) for mtree which contains the path /data/col1/rl_test/testfile. Retention lock cannot be disabled for an MTree using compliance mode: sysadmin@ddxxxx# mtree retention-lock disable mtree /data/col1/rl_test_comp **** Operation is not allowed because the system is a retention-lock compliance system Can replication still be used against MTrees which have retention lock enabled? Yes, retention locked MTrees or files can be replicated using various replication topologies: Directory replication - only supported for files using governance mode - does not replicate minimum and maximum retention periods to the destination system.MTree replication - can be used for either governance or compliance mode data and does replicate minimum and maximum retention periods to the destination system.Collection replication - can be used for either governance or compliance mode data and does replicate minimum and maximum retention periods to the destination system. Note: For retention locks to be preserved on destination systems: Both the source and destination systems must have corresponding retention lock licenses that are installed.If replicating a RLC enabled Mtree then both source and destination DDs must have RLC configured or the next error will be received:"A compliance retention-locked MTree cannot be replicated to a destination that is not enabled for compliance retention-lock."MTree replication contexts must have 'Replication propagate-retention-lock' set to Enabled.For files replicated by directory replication, the corresponding MTrees minimum and maximum retention periods must be manually set on the destination system. Are there any other restrictions when replicating retention locked files? Yes, resyncs of replication contexts (that is trying to reestablish a previously configured but broken context) where retention locked data is used to can fail. Note: If MTree replication is used, and the destination MTree contains retention-locked files which do not exist on the source, a resync fails.If directory replication is used to and the destination has a retention lock that is enabled but the source does not, then a resync fails. Note: When using MTree replication, a resync succeeds in the following scenarios, if the destination MTree does not contain retention-locked files not present on the source DDR: The source MTree does not have retention lock enabled, but the destination does.The destination MTree does not have retention lock enabled, but the source does. It is also not possible to enable retention lock compliance mode on an MTree which is already a member of an MTree replication context. In this scenario: The MTree replication context should be broken on both source and destination systems: # replication break mtree://[destination system]/data/col1/[mtree] A new MTree replication context should be created on both source and destination systems: # replication add source mtree://[source system]/data/col1/[mtree] destination mtree://[destination system/data/col1/[mtree] Retention lock compliance mode should be enabled on the source system: # mtree retention-lock enable mode compliance mtree [mtree] The newly created replication context should be resynced on the source system: # replication resync mtree://[destination system/data/col1/[mtree] Can retention-locked files be fast copied? Yes, files which are retention locked can be fast copied as normal. If the destination MTree holding the fast copy is retention lock enabled, then the retention lock of the file is preserved against the fast copy. If the destination MTree is not retention lock enabled, then the fast copy is not retention locked. Are there any other restrictions in system functionality when using retention lock? The following commands are disallowed on systems using retention lock compliance mode: # user reset # filesys destroy # filesys archive unit del [archive unit name] Such systems cannot be booted to single user mode for recovery by technical support without use of a USB drive and physical access to the system.The following commands are disallowed against MTrees using retention lock compliance mode: # mtree delete [mtree] # mtree retention-lock reset [min-retention-period period | max-retention-period period] mtree [mtree] # mtree retention-lock disable mtree [mtree] # mtree retention-lock revert However, in DDOS 7.3 and later releases, a provision has been made for customers to be able to delete Retention Lock Compliance enabled MTrees if: DD is running DDOS 7.3 or later.The RLCE MTree to be deleted is empty (has zero files and directories).Administrator successfully passes security-officer authentication. In systems configured with Long-Term Retention (Cloud Tier), similarly destructive commands may also be disallowed, for example: # cloud unit del cloud unit name> Note: MTrees using retention lock governance mode (or where this mode was once enabled) can only be deleted once the MTree contains no files - if there are any files remaining in the MTree, an error is returned. Is the system clock important on retention lock enabled systems? Yes, retention lock compliance-enabled systems enable an internal 'security clock' to prevent malicious tampering with the system clock (which may allow retention-locked files to be deleted early). The times of the security and system clocks are regularly compared and if there is an accumulated 2-week skew between the two in a single calendar year the Data Domain File System (DDFS) is automatically disabled to prevent access to data on the DDR. This can also happen if the system clock is suddenly modified and the time is changed by more than 2 weeks.In this scenario DDFS can be reenabled by: Logging into the DDRCheck that the system clock is set correctly.Enabling the file system: # filesys enable When prompted enter details of a user with the role of security to allow the security clock to be reset, and enable DDFS. Can the system clock be synchronized with Active Directory on retention lock compliance-enabled systems?No, when retention lock compliance is enabled, CIFS servers no longer synchronize the system time with Active Directory. If there is a time difference of greater than five minutes between the system and Active Directory, the CIFS server displays an error message when an Active Directory user attempts to log in, or the system attempts to join an Active Directory domain. Configure Active Directory time with NTP to avoid this error.This is in contrast to systems where retention lock compliance is not enabled but Active Directory is in use. In this situation, it is not recommended to enable NTP as there may be time setting conflicts between Active Directory and NTP.What steps can be taken if a DDR using retention-locked files fills to capacity? Assuming there is no 'cleanable' space on the DDR (clean is run but the system remains filled to capacity) its contents should be reviewed to determine if: There are any files which are not retention locked which can be removed. There are any files that are locked using governance mode which can have their locks reverted and removed. Once this is done, clean should be run again to physically free space on the system. Alternatively, if no physical data can be deleted from the system, additional physical storage should be added to the DDR and the file system expanded (assuming expansion is supported by the current DDR or configuration).If the only files on the system are locked using compliance mode, it is not possible to revert the locks and remove these files. As a result, space is not able to be freed unless: The retention period is reached for some or all the files. After this point, they can be deleted and clean run (as described above). The system is reinstalled from a USB drive (which causes loss of all data on the DDR). More physical storage is added to the system (as described above). Note: It is entirely possible to completely fill a DDR with files locked using compliance mode. In this scenario, there is nothing an administrator or technical support can do to free space (that is there is no low-level functionality to remove/revert compliance mode locks and delete corresponding files early).

Support Cases