Symptoms

When PowerStore encounters an issue, an error alert is generated to help identify the issue. This article explains the different possible causes and the relevant remediation for these error alerts. If your issue is not resolved despite the remediation that is provided, check the Dell Support contact options. Note: Cannot find the information that you need? Let us know using the feedback form at the bottom of this article.

Cause

When an appliance reaches its full capacity, it enters a condition that is known as Write Protect Mode (WPM). In WPM, all volumes transition to read-only, and a repair of the system is required.For an explanation of how Physical Capacity is calculated, see article 000188491: PowerStore: How PowerStore Physical Capacity is calculated.

Resolution

Alert IDDATA_PATH_WRITE_PROTECT_MODE_ENTERDATA_PATH_APP_EXIT_WPM_WITH_ENOUGH_MD_NOTIFYDATA_PATH_APP_WPM_SHORT_OF_MD_NOTIFYAlert TextThe <user or system > data capacity has reached its full capacity.Appliance has exited the write protection state with 16 GB of free capacity.Appliance cannot exit the Write Protection mode due to insufficient system data capacity.Error Code(0x00200701)(0x00201401)(0x00201501)ResolutionWhat happens when in Write Protect Mode (WPM)?When an appliance reaches its full capacity, it enters a condition that is known as Write Protect Mode (WPM), where all volumes transition to read-only. The alert 0x00B00702 (alert text: Service level is degraded, Management functions are limited, and I/O is impacted) is raised.Any Write I/O failsOperations such as Snapshots and Replication may be disabled.All metrics (capacity and performance) are unavailable. In PowerStore 3.0, and later, the metric table is updated while in WPM. Recommendation: The system should not exceed the 85% threshold. Cause: The PowerStore Total Physical Capacity consists of both User Data and System Data.On PowerStoreOS 3.0 and later, the following is displayed on PowerStore Manager. The user is presented with a more precise breakdown of both user data and system data: Figure 1. Physical Capacity Breakdown On PowerStoreOS pre-3.0 versions, the following is displayed on PowerStore Manager. The user is only presented with a breakdown of Physical Used Capacity: Figure 2. Physical Usage When the system reaches 70% capacity, alerts are raised to warn of lack of space. See article 000123351 PowerStore Alerts: Capacity Utilization. If the user takes no action, and the system reaches 100%, the system enters Write Protect Mode (WPM), which indicates that all volumes enter read-only mode, and the alert 0x00200701 is raised. Alert 0x00200701 indicates a space issue with either User Data or System Data.If a node is down for a prolonged period, it may lead to physical capacity increase. How to determine the WPM root cause:Users can determine the WPM condition root cause based on the alert text: User Data - (0x00200701) The user data has reached its full capacity.System Data - (0x00200701) The system data has reached its full capacity. The workaround is determined by the type of WPM condition (User Data or System Data). Solution: Recovering from an OOS condition depends on the reason for the OOS. The cause of the WPM condition is contained in the alert text: User Data - (0x00200701) The user data has reached its full capacity.System Data - (0x00200701) The system data has reached its full capacity. The workaround is determined by the type of WPM condition (User Data or System Data). User DataUse one of the following options: Add Drives (this is the recommended option): Consult with your account team to analyze your capacity requirements and add more drives.Delete Volumes and or Snapshots. If the user requires information about which volumes or snapshots to delete, escalate to Global Services for guidance. Additional information Before you carry out any actions, you should do the following: Stop all I/O to the array (disable FC/iSCSI ports on the switch) to avoid experiencing WPM again. Pause all snapshot schedules, Replication, and any external automation related to volume management. To get out of WPM condition, the appliance must reclaim at minimum 16 GB of space. To do so, the user must delete a volume with at minimum 16 GB of unique data. If the user manages to reclaim such space, and there is sufficient space for System Data, alert 0x00200701 (Alert text: The user data has reached its full capacity) clears, and alert 0x00201401 (Alert text: Appliance has exited the write protection state with 16 GB of free capacity) appears. Once in this state, volumes transition to read/write access, and the user should continue reclaiming space until the appliance reaches 85% Used Physical Capacity. It is highly advised deleting volumes or snapshots one at a time. See the specific Operating System manual for instructions how to reclaim space from host side. Once at 85% Used Physical Capacity, the user must manually clear the alert 0x00201401. Note: Reclaiming space by deleting snapshots or volumes initially requires additional space for System Data, and hence, may lead to alert 0x00201501 (Alert text: Appliance cannot exit the Write Protection mode due to insufficient system data capacity). PowerStoreOS 1.0.3 introduced a new service script (svc_volume_space_metrics) that allows users to view additional information about volume space statistics when an appliance is in WPM. For more information, see the How to use svc_volume_space_metrics section later in this article. System Data Use one of the following options: Add Drives (this is the recommended option): Consult with your account team to analyze your capacity requirements and add more drives. If the user cannot add drives, escalate to technical support for assistance. This information can be viewed from the Volume Capacity Dashboard. In this example, deleting the below volume reclaims 59.8 GB of space that can be used for either User Data or System Data. System Data requires a minimum of around 100 GB of unique space to be able to expand. Sometimes it may require 200 GB of space to properly expand. Figure 3. Usage Additional information Usually, when an appliance enters WPM due to System Data, User Data also runs out of space. This is because there is no capacity that is left for expanding (either System Data or User Data). To get out of WPM, typically the user is required to add additional drives and scale up the appliance. When the system raises the alert 0x00200701 (Alert text: The System capacity has reached its full capacity.) This indicates that System Data has run out of space and cannot be expanded. Avoid deleting volumes in this condition and escalate to Global Services for support. How to use svc_volume_space_metrics PowerStoreOS 1.0.3 introduced a service script that allows users when in WPM to see additional information about each volume tree. To run the script, SSH must be enabled on the specified appliance, and the user must connect using SSH to the cluster IP using the service account. When trying to run this service script when not in WPM, the service script fails. When appliance runs out of space, the user can run the script and see additional information, such as:Family Unique Physical Used - This represents the unique data (compressed) this volume, and all its family (snapshots and clones) releases if this volume is deleted.Example: [SVC:service@xxxxxx-A user]$ svc_volume_space_metrics 2020-10-29 12:28:33,997 - INFO - event_client init() address: localhost node: A 2020-10-29 12:28:34,248 - INFO - PyCycloneDp.init localhost 2020-10-29 12:28:34,318 - PyCycloneDp - INFO - logging in PyCycloneDp.__init__ Namespace state is ro-oos Retrieve family space stats from DP namespace. It will take some time... 2020-10-29 12:28:35,972 - INFO - event_client init() address: localhost node: A 2020-10-29 12:28:36,175 - INFO - PyCycloneDp.init localhost 2020-10-29 12:28:36,220 - PyCycloneDp - INFO - logging in PyCycloneDp.__init__ --------------------------------------+------------------------------------------------------+-------------+----------------+----------------+-----------+---------- Family ID | Primary Volume/FS/vVol Name | Type | Family Unique | Family Shared | No. of | No. of | | | Physical Used | Logical Used | Clones | Snapshots --------------------------------------+------------------------------------------------------+-------------+----------------+----------------+-----------+---------- d301959c-6eb9-4e23-967a-e0b3cdf06bea | labtea-vol-1 |Block | 97.91 GB| 486.36 GB| 0 | 1 6c97fdd8-5b01-44ef-adaf-047449f83c27 | labtea-vol-2 |Block | 99.47 GB| 492.30 GB| 0 | 1 Alert ID DATA_PATH_WRITE_PROTECT_MODE_EXTREMELY_HIGH Alert Text The space utilization of % (tier_name) is extremely high. Error Code 0x00200703 Impact Several internal services (such as NAS and UI) may not operate optimally. Resolution Cause: Some of the appliance capacity is reserved for internal services (such as NAS core services, Data Collects, internal DB and more). When the appliance reaches Write Protect Mode (0x00200701), the user can no longer write any data, however internal services can still access critical information (only from PowerStoreOS 3.0), once the internal reserved capacity also runs out, this alert is raised, CP, and SDNAS (and other services) may not operate optimally. Solution: Contact Dell Technical Support for immediate assistance.

Support Cases