Symptoms

How long should it take for symcfg -env_data, to detect hardware failures?An email alert is sent out for a hardware failure. Going to the System Dashboard > Hardware status, it does not immediately show any failed status icons.What needs to be configured in Unisphere for VMAX, to show failed hardware after receiving an email alert?Power loss to an array does not get picked up until 20 minutes later, using "symcfg -sid xxx -env_data list" command.For example, a power outage occurs:# symcfg -sid xxxx -env_data list -v Symmetrix ID : 000xxxxxxxTimestamp of Status Data : 08/25/2016 10:51:17 System Bay Bay Name : SB-1 Bay LED state : Normal (On) Front Door Bay LED state : Normal (On) Number of Standby Power Supplies : 6 Number of Drive Enclosures : 4 Number of Enclosure Slots : 2 Number of MIBE Enclosures : 2 Status of Contained Modules Standby Power Supplies SPS-1A (Aggregate) : Normal SPS-TRAY-1A : Normal SPS-BATTERY-1A : Normal SPS-1B (Aggregate) : Normal SPS-TRAY-1B : Normal SPS-BATTERY-1B : Normal SPS-2A (Aggregate) : Normal SPS-TRAY-2A : Normal SPS-BATTERY-2A : Normal SPS-2B (Aggregate) : Normal SPS-TRAY-2B : Normal SPS-BATTERY-2B : Normal SPS-3A (Aggregate) : Normal SPS-TRAY-3A : Normal SPS-BATTERY-3A : Normal SPS-3B (Aggregate) : Normal SPS-TRAY-3B : Normal SPS-BATTERY-3B : Normal An alert is sent out via email: Description: Power system AC line interruption detected. Severity: WARNING State: NEW Managed Object: 000xxxxxxxxx Server Timestamp: Thu Aug 25 10:55:50 EDT 2016 Alert ID: 9f65c4a4-665c-48cb-9cf6-76a215d9058f Symmetrix: 000xxxxxxx Application: SMC Policy Name: Environmental Alert Error Code: 0x001d Event Code: 1077 SMAS Server Name: xxxxxxx SMAS Server Canonical Name: xxxxxx >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> symevent shows: Detection time Dir Src Category Severity Error Num------------------------ ------ ---- ------------- ------------ ----------Thu Aug 25 10:55:48 2016 IM-2A Symm Environment Warning 0x001d A Symmetrix power subsystem AC line interruption was detectedThu Aug 25 10:55:49 2016 IM-1A Symm Environment Warning 0x001d A Symmetrix power subsystem AC line interruption was detectedThu Aug 25 10:55:49 2016 IM-4A Symm Environment Warning 0x001d A Symmetrix power subsystem AC line interruption was detectedThu Aug 25 10:55:50 2016 IM-3A Symm Environment Warning 0x001d A Symmetrix power subsystem AC line interruption was detected storeventd shows: [1796 SymmPoller] Aug-20 16:14:55.427 : [evtd_symmEnvHelper] The module [SB-1/SPS-2A (Aggregate)] state is not available(nothing for the power loss at 10:51) [1796 SymmPoller] Aug-25 12:53:25.095 : [evtd_symmEnvHelper] The module [SB-1/SPS-1B (Aggregate)] state is not available [1796 SymmPoller] Aug-25 12:53:25.095 : [evtd_symmEnvHelper] The module [SB-1/SPS-2B (Aggregate)] state is not available [1796 SymmPoller] Aug-25 12:53:25.095 : [evtd_symmEnvHelper] The module [SB-1/SPS-3B (Aggregate)] state is not available [1796 SymmPoller] Aug-25 16:56:20.738 : [evtd_symmEnvHelper] The module [SB-1/SPS-2B (Aggregate)] state is not available There is no indication in the GUI, of a failure: Ran a "symcfg discover" to see if this changed anything:symapi.log shows: 08/25/2016 10:58:34.808 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 10:58:35.401 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:03:05.190 5936 5524 EMC:SYMCFG SymDiscover() Function start08/25/2016 11:03:11.024 5936 5524 EMC:SYMCFG SymDiscover() Function complete.08/25/2016 11:08:24.901 3696 4724 EMC:SMBASE qos_call_status QOS 'SymQosGet' operation SUCCEEDED.08/25/2016 11:11:23.979 372 5540 EMC:SYMCFG SymDiscover() Function start08/25/2016 11:11:31.093 372 5540 EMC:SYMCFG SymDiscover() Function complete.08/25/2016 11:11:51.420 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:11:51.420 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:11:52.044 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:41:13.771 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:41:14.223 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:44:15.185 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon)08/25/2016 11:44:15.497 3696 2040 EMC:SMBASE iSymDaemonControl daemon: 'storevntd', action: 2, rqst: 'snmp_host' ([PDS/Daemon] The request failed at the daemon) The first "Failed" detection is 20 minutes later:# symcfg -sid xxxx -env_data list -v Symmetrix ID : 000xxxxxxxxxTimestamp of Status Data : 08/25/2016 11:11:23 System Bay Bay Name : SB-1 Bay LED state : Normal (On) Front Door Bay LED state : Normal (On) Number of Standby Power Supplies : 6 Number of Drive Enclosures : 4 Number of Enclosure Slots : 2 Number of MIBE Enclosures : 2 Status of Contained Modules Standby Power Supplies SPS-1A (Aggregate) : Normal SPS-TRAY-1A : Normal SPS-BATTERY-1A : Normal SPS-1B (Aggregate) : Failed SPS-TRAY-1B : Failed SPS-BATTERY-1B : Failed SPS-2A (Aggregate) : Normal SPS-TRAY-2A : Normal SPS-BATTERY-2A : Normal SPS-2B (Aggregate) : Failed SPS-TRAY-2B : Failed SPS-BATTERY-2B : Failed SPS-3A (Aggregate) : Normal SPS-TRAY-3A : Normal SPS-BATTERY-3A : Normal SPS-3B (Aggregate) : Failed SPS-TRAY-3B : Failed SPS-BATTERY-3B : Failed Immediate following, I performed a "partial" GUI refresh that still did not change system hardware status. To get the system hardware to show red X's, a "full" refresh is needed:No restarting of SMAS is required.Once power is restored: Symmetrix ID : 000xxxxxxxTimestamp of Status Data : 08/25/2016 12:49:40 System Bay Bay Name : SB-1 Bay LED state : Normal (On) Front Door Bay LED state : Normal (On) Number of Standby Power Supplies : 6 Number of Drive Enclosures : 4 Number of Enclosure Slots : 2 Number of MIBE Enclosures : 2 Status of Contained Modules Standby Power Supplies SPS-1A (Aggregate) : Normal SPS-TRAY-1A : Normal SPS-BATTERY-1A : Normal SPS-1B (Aggregate) : N/A SPS-TRAY-1B : N/A SPS-BATTERY-1B : N/A SPS-2A (Aggregate) : Normal SPS-TRAY-2A : Normal SPS-BATTERY-2A : Normal SPS-2B (Aggregate) : N/A SPS-TRAY-2B : N/A SPS-BATTERY-2B : N/A SPS-3A (Aggregate) : Normal SPS-TRAY-3A : Normal SPS-BATTERY-3A : Normal SPS-3B (Aggregate) : N/A SPS-TRAY-3B : N/A SPS-BATTERY-3B : N/ASymmetrix ID : 000xxxxxxxxTimestamp of Status Data : 08/25/2016 13:09:47 System Bay Bay Name : SB-1 Bay LED state : Normal (On) Front Door Bay LED state : Normal (On) Number of Standby Power Supplies : 6 Number of Drive Enclosures : 4 Number of Enclosure Slots : 2 Number of MIBE Enclosures : 2 Status of Contained Modules Standby Power Supplies SPS-1A (Aggregate) : Normal SPS-TRAY-1A : Normal SPS-BATTERY-1A : Normal SPS-1B (Aggregate) : Normal SPS-TRAY-1B : Normal SPS-BATTERY-1B : Normal SPS-2A (Aggregate) : Normal SPS-TRAY-2A : Normal SPS-BATTERY-2A : Normal SPS-2B (Aggregate) : Normal SPS-TRAY-2B : Normal SPS-BATTERY-2B : Normal SPS-3A (Aggregate) : Normal SPS-TRAY-3A : Normal SPS-BATTERY-3A : Normal SPS-3B (Aggregate) : Normal SPS-TRAY-3B : Normal SPS-BATTERY-3B : Normal

Cause

There is no issue here. The data that the symcfg -env_data reports on is coming from SymmWin. When a power loss occurs, SymmWin is caching the FRU status and will refresh it (if requested) only once every 20 minutes.The reason is that reading the status frequently puts a high load on the Enginuity / HYPERMAX OS and may affect performance.Power events should be reported by Enginuity / HYPERMAX OS events / errors and not Unisphere.

Resolution

The Dell EMC product is working as designed.There are other power related events that the Solutions Enabler daemon and Unisphere for VMAX can be configured to report on. These specific events are covered In the installation and configuration guides for these products.

Support Cases