Symptoms

PowerProtect DDOS includes a safeguard that disables the DD File System (DDFS) If 1: The system time jumps back by more than 60 s. If such a change in system time occurs,, and/OR 2: If any other hardware components fails and system sets a critical environmental alert for enclosure 1 The following takes place: DDFS is disabled and does not automatically restartAn alert (EVT-ENVIRONMENT-00052) is posted, that is: Event posted: p0-32 -EVT-ENVIRONMENT-00052: File system is disabled due to a critical condition.EVT-OBJ::Enclosure=1 EVT-INFO::Cause=System Time backward jumped Event posted: p0-32 -EVT-ENVIRONMENT-00052: File system is disabled due to a critical condition.EVT-OBJ::Enclosure=1 EVT-INFO::Cause=System has Critical alert Once this issue is encountered: DDFS is unable to manually restart (it panics during boot)Restoring the date and time (to reverse the backwards jump) does not allow DDFS to startResolving the hardware problem does not allow DDFS to start

Cause

Data domain filesystem can not be enabled until we fix the critical environment condition just to safeguard the DD and other application integrated to DD This safeguard was implemented as a backwards jump in system time may adversely affect certain backup applications which store data on the DDR. As a result, it is designed such that the administrator of the DDR has to allow the change in system time before DDFS can be re-enabled. Note: Once this issue is encountered, DDFS cannot be enabled as it refuses to start, for example: # filesys enable Please wait... 01/01 20:32:10.217 (tid 0xxxxxxx): INFO: Event posted: m0-28 (2100001c:553648156): EVT-FILESYS-00008: Filesystem has encountered an error and is restarting. **** There was a problem bringing up the filesystem. Status: The filesystem is aborting due to a problem. In addition, reversing the backwards jump in system time does not allow DDFS to be re-enabled (the issue persists).

Resolution

To enable DDFS, follow these steps. If the affected DD is the active node in a DD HA pair, apply these steps to both nodes before enabling the File System (FS). From the UI: 1. Ensure the system’s date, and time are correct. If you change the time zone, the DDR might prompt for a reboot. Perform this reboot immediately to ensure that all processes recognize the new time zone. Go to Administration > Settings > TIME AND DATE If you have NTP enabled, disable it temporarily to correct the clock. Go to MORE TASKS > Configure Time Settings Then configure the time settings again to re-enable NTP if needed 2. Clear the emergency alert corresponding to the 'filesystem disabled due to a critical condition' error. Go to Health > Alerts > CURRENT ALERTS Wait one minute for the alert to clear and the system status to update. If you do not, the system status might not fully update before the DDFS process starts, potentially causing a one-time FS crash and alert. 3. Enable the filesystem if it is not already enabled after clearing the alert. Go to Data Management > File System > Summary > Click Enable, towards bottom of the page. DDFS should now boot/run as normal. If you failed to wait a sufficient time after clearing the alert before starting the FS process, you may receive an alert on the CLI about the FS to have encountered a problem, however, FS will continue trying to start up and, if the problem was as described in this KB, the FS process will eventually enable. From the CLI: 1a. Ensure the system’s date, and time are correct. If you change the time zone, the DDR might prompt for a reboot. Perform this reboot immediately to ensure that all processes recognize the new time zone. 1b. Ensure all the hardware components in DD are in good state 2. Clear the emergency alert corresponding to the 'filesystem disabled due to a critical condition' error: # alert clear alert-id [alert id] For example, if this were alert p0-32 (as shown above): # alert clear alert-id p0-32 3. Wait one minute for the alert to clear and the system status to update. If you do not, the system status might not fully update before the FS process starts, potentially causing a one-time FS crash and alert. 4. Now DDFS should automatically come online, you may check the status of filesystem with "# filesys status". if not, Enable DDFS: # filesys enable DDFS should now boot/run as normal. If you failed to wait a sufficient time after clearing the alert before starting the FS process, you may receive an alert on the CLI about the FS to have encountered a problem, however, FS will continue trying to start up and, if the problem was as described in this KB, the FS process will eventually enable. For further information about this safeguard or any of the information contained within this article, please contact Dell Technical Support.

Support Cases