Symptoms

Symptoms include: Support Materials or support Data Collections (DC) fail to run, or get stuck in running state in the PowerStore Manager user interface (UI). Running DCs over CLI (svc_dc run) fails with: FAILURE, Running too longTimed out receiving service data bundle command response from container Volumes display capacity of 0GB. Unable to create or delete any objects (such as volumes, and so on). Volumes cannot be expanded - error 0xE0A080030019.Protection policies no longer working (new snapshots are not created and old ones not expiring), and replication verification fails.Unable or view or change the SSH status. Error: There was an error retrieving this information. Unknown property is_ssh_enabled requested. (0xE04040020002)LDAP domain connection errorsAlerts for root partition full or running out of space. Temp DC files are not cleaned up from /cyc_var/cyc_service/tmp on the secondary node, possibly leading to a full root partition. The system generates warnings: Root partition usage of node X has exceeded Y% (codes: 0x00400601 or 0x00400602)In some extreme cases where no action is taken promptly to resolve the root partition space issues, the secondary node may go into service mode.Monitoring > System Checks > Run System Check fails with Fireman command failed. (0xE0F010200004) Example of the DC issue as seen from PowerStore Manager: Example of the system check failure as seen from PowerStore Manager:

Cause

Many leaked systemd login sessions lead to a memory leak in the service container. The fireman service in the service container is killed during DC collection due to a out of memory condition.The fireman service remains down if system does not detect its failure after the fireman service is killed.

Resolution

Fix The fix to prevent this issue from happening is in PowerStoreOS 2.1.1.0-1649887. The recommendation is to upgrade.This fix is listed in the PowerStoreOS version 2.1.1.0 release notes revision A03 or later: Issue IDFunctional AreaDescriptionMDT-361718 PowerStore Manager(GUI)Due to an issue with the Control Path or Management resources, a degradation of the PowerStore system user interfaces may occur over time. The degradation may cause a slow response or the inability for PowerStore Manager (UI) to collect data. Workaround Note: This workaround is for PowerStore T models only. PowerStore X has more requirements and steps that must be performed before restarting any services. If the problem is already present, contact Dell Technical Support or your Authorized Service Representative, and quote this Knowledgebase article ID, before attempting to upgrade.In order to resolve this issue when already present, two services must be restarted. Service container on the affected node. The service container takes around 10 minutes to restart.No impact on the system other than a brief loss of access to the service container (ssh or cli) of the affected node. Control Path (CP) or management services. Takes around five minutes to restart.No impact on the system other than a brief loss of access to the PowerStore Manager user interface. After a few minutes, the space on the secondary node's root partition drops down to normal levels.If the /cyc_cfs partition is above 85%, delete old DCs from the PowerStore Manager user interface.You may see some alerts after restarting the services, such as: SupportAssist connectivity alerts.Replication RPO not met alerts.Snapshot automatic deletion alerts. These should all clear out by themselves after the action plan. Give it enough time as some may need to wait until the next RPO cycle (replication) or snap schedule to kick in.

Support Cases