Symptoms

Received one or more of the following Field Replaceable Units (FRU) notifications in the email alert or Dial Home (DH) message:Uninterruptible-Power-Supplies (UPS-A/B) for the VS2 or VS6 and the data that is seen in the alert or DH message where you see "CDATA" ups@UPS-x: PartNo 078-000-052 / SerialNo xxxxxxxxxxxxxx /RevNo xxxx: The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x, xx.xx.x.x, xx.x.xx}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. ups@UPS-x: PartNo 078-000-079 / SerialNo xxxxxxxxxxxxx /RevNo xxxx: The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.xx.x, xx.xx.x.x, xx.xx.x.xx}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. Note: There are several serial numbers that may be seen in the alert or DH message. If a UPS starts with any of the serial numbers mentioned below, dispatch the service request (SR), with work order, for the UPS replacement. A-Older UPS: APDxxxxxxxxxxxx APMxxxxxxxxxxx ASxxxxxxxxxx EMPxxxxxxxxxxx ENP0013xxxxxxx till ENP0021xxxxxxx PITxxxxxxxxxxx PTCKxxxxxxxxxx How to read the UPS serial numbers to know what year a battery was installed, any battery installed before 2021 is over five (5) years old and needs to be replaced. For those serial numbers starting with APD, EMP or ENP ignore the first five (5) places, then the next four places equal the year and week of that year a battery was installed into the UPS, e.g. APD00083010132 - ignore APD00, then the next four (4) digits, 0830, 08 is the year 2008, and 30 is the 30th week of 2008 the battery was installed, so the fourth week of July 2008. e.g. EMP00160100327 - ignore EMP00, then the next four (4) digits, 1601, 16 is the year 2016, and 01 is the 1st week of 2016 the battery was installed, so the first week of January 2016. e.g. ENP00141500271 - ignore ENP00, then the next four (4) digits, 1415, 14 is the year 2014, and 15 is the 15th week of 2014 the battery was installed, so the first full week of April 2014. For those serial numbers starting with AS ignore the first two (2) places, then the next four places equal the year and week of that year a battery was installed into the UPS, e.g. AS1839290999 - ignore the AS, then the next four (4) places, 1839, 18 is the year 2018, and the 39 is the 39th week of 2018, fourth week of September 2018. For those UPS serial numbers starting with PITFRA and PTCK replace them, those batteries are all well over five (5) years old. Note: If any UPS serial number starts with ENP0020xxxxxxx and higher, DO NOT DISPATCH the SR for a UPS replacement, unless the UPS is reporting as failure or not charging, then replace. Note: While investigating any VPLEX UPS issue, check the serial numbers of all UPS in the configuration. If any have the serial numbers listed in the "Older UPS" listing, they too are to be replaced. Dispatch the SR, with work order for the UPS replacement, and order part number 078-000-079 for the replacement UPS. If more than one UPS is showing with an older serial number, then they are to be replaced as well. Be sure to state that when more than one UPS is to be replaced, that they are to be replaced one at a time and allow the VPLEX to stabilize, so as not to take down the entire VPLEX. If DH SRs or alerts are seen reporting with in a 30-day period for the same VPLEX, and the UPS serial number starts with ENP0020, or lower, it may be a sign that the batteries are starting to have problems charging or holding a charge and may require being replaced.If three or more UPS DH/alerts are reported with in a two-week period for the same VPLEX, same serial number, follow the actions to dispatch the UPS for replacement. If there are any questions on this, reach out to the Remote Support team for further assistance. Note: For the replacement UPS, only order part number 078-000-079 ==================================================Standby-Power-Supplies-A or B (SPS-A or SPS-B) - VS2 Only: sps@stand-by-power-supply-A: PartNo 078-000-050 /SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x.x, xx.xx.x.x, xx.xx.x.x}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. sps@stand-by-power-supply-B: PartNo 078-000-050 /SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x.x, xx.xx.x.x, xx.xx.x.x}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. Sometimes a remote cluster is reporting on its peer: peer-sps@stand-by-power-supply-A: :The specified FRU Faulted. [Versions:MS{xx.xx.x.xx.x, xx.xx.x.x, xx.xx.x.xx}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The specified FRU is not present. This latter DH or alert maybe due to a communication issue between clusters in a Metro configuration is down. The VPN must be checked, and if it is down try restarting it with vpn start. If this does not work, check the certificates to see if they may have expired. See article 000154841, VPLEX: How to check if Host Certificates are still valid (Log in as a registered user to Dell Support is required to view article). If a certificate shows expired, see article 000169421, VPLEX: How to manually re-create the security certificates after expiring, forgetting the passphrase, the VPN certificates must be re-created, or getting the SSL exception error (Log in as a registered user to Dell Support is required to view article)==================================================Power Supplies (ps) ps a# or ps b# or power-supply-a# or b#-VS2:Samples of how these issues may be seen in the DH or alert messages:Power-Supply-a0: /engine/psa0: PartNo 071-000-543 / SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x.x, xx.xx.x.x, xx.xx.x.x}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{x.x.x.x}] RCA: The state of the specified FRU is faulted. Power-Supply-a1: /engine/psa1: PartNo 071-000-543 / SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x.x, xx.xx.x.x, xx.xx.x.x}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. Power-Supply-b0: /engine/psb0: PartNo 071-000-548 / SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.xx.x.x.x, xx.xx.x.x, xx.xx.x.x}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. Power-Supply-b1: /engine/psb1: PartNo 071-000-543 /SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state the of specified FRU is Faulted. [Versions:MS{xx.xx.x.xx.x, xx.xx.x.x, xx.xx.x.xx}, Director{x.x.xxx.xx.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: : The state of the specified FRU is faulted. Power-Supplies (psa# or psb# or power-supply-a# or b#) VS6:A sample of how this issues may be seen in the DH or alert messages for a VS6: /engine/psa/b#: PartNo 071-000-611-01 /SerialNo xxxxxxxxxxxxxx /RevNo xxxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.x.x.x.x, xx.x.x.x.x, xx.x.x.x.x}, Director{x.x.x.x.x}, ClusterWitnessServer{xx.xx.x.x}] RCA: The state of the specified FRU is faulted. Note: Power-Supplies are tied to the SPS' on the VS2 and BBUs on the VS6. If you are seeing a lot of power-supply DHs or alerts, check the status of the SPS on the VS2 and BBUs on the VS6. =================================Battery Backup Unit (BBU) a# or b#-VS6:If you see a call home/alert for the BBUs on the VS6 only, see article 531093, "VPLEX: BBUs are frequently cycling from "Charging" to "Fully-Charged" for more details.A sample of how this issues may be seen in the DH or alert messages for a VS6: /engine/bbu-xx: PartNo 078-000-123-05 /SerialNo xxxxxxxxxxxxxx /RevNo xxx : The operational state of the specified fru is Faulted. [Versions:MS{xx.x.x.x.x, xx.x.x.x, xx.x.x.x}, Director{x.x.x.x.x}, ClusterWitnessServer{xx.x.x.x}] RCA: The state of the specified FRU is faulted.

Cause

The alerts or DH messages listed in the Issue section may be due to battery tests or checks by those components. The FRU monitoring portion of the code is not aware of these battery tests or checks by the components. The code only knows when it sees what it knows as an error and reports the event using the call home notification mechanism. This sends the end user an email alert and sends a Dial Home report back to Dell.

Resolution

The end user should be checking and confirming the state of any FRU reported as "faulted" covered in this article. Dell is also made aware of FRU Faulted events as the VPLEX reports these events back to Dell. SRs are opened for Dell to investigate the issues and let the customer know where there is an issue reported.Steps for checking the state of reported FRU:The end user must to log in to the VPLEX when any of the alerts listed in the "Issue Section" of this article have been reported and check the status of the reported FRU. If the FRU the issue was reported for is OK, the alert can be ignored. If, when checking the status of a FRU, it shows being in a state other than indicated in the sample outputs, you must contact Dell Customer Support to report the issue.Follow the steps below to check the state of the FRU mentioned in this article.To determine which cluster and component the dial home (DH) event/alert was reported from follow the steps below. In the alert message, to find out what cluster and component the event was reported for, look for the Sub-ComponentID line the issue was reported for, the first number listed is the cluster ID. Example of call home: SymptomCode: 0x8a4861d7 Category: Status Severity: Error Status: Failed Component: DIRECTOR <-------see info for ComponentID ComponentID: director-2-2-B < <cluster-2, director-2-2-B was where the issue occurred SubComponent: ZPEM SubComponentID: - CallHome: Yes FirstTime: 2022-09-29T15:27:35.096Z LastTime: 2022-04-13T17:26:48.047Z Count: 4 CDATA: ups@UPS-B: PartNo 078-000-052 /SerialNo ABCxxxxxxxxxx /RevNo 1XE1 :The operational state of the specified fru is Faulted. [Versions:MS{x.x.x.x, x.x.x.x, x.x.x.x}, Director{x.x.x.x}, ClusterWitnessServer{x.x.x.x}] RCA: The state of the specified FRU is faulted. Note: For UPS' reported in the FRU Faulted alert CDATA area be sure to check the serial numbers per the list in the Issue section. Even UPS' not reported for a FRU Faulted alert.If any UPS has an older serial number, have those UPS replaced as the batteries have reached the end of their operational state and may start to report failures. By having older UPS' replaced, before an issue is reported, this prevents any type of outage of the VPLEX due to a dual UPS battery failure. Log in to the VPLEX Log in to the management server for the VPLEX cluster the DH event was reported from: Sample output: login as: service Using keyboard-interactive authentication. Password: Last login: Wed Apr 13 18:05:04 2016 from xx.xx.xx.xx service@ManagementServer:~> Next, access the VPlexcli Sample output:For pre-6.2.x: service@ManagementServer:~> vplexcli Trying ::1... Connected to localhost. Escape character is '^]'. Enter User Name: service Password: Creating logfile:/var/log/VPlex/cli/session.log_service_localhost_ Logfile_T24531_YYYYMMDDHHMMSS VPlexcli:/> For 6.2.x: service@ManagementServer:~> vplexcli Trying ::1... Connected to localhost. Escape character is '^]'. VPlexcli:/> Below are the steps for how to check the state of each FRU that this call home could be for.Uninterruptible Power Supply (UPS) - VS2 or VS6Follow the steps to log in as listed above following steps 2 or 3. Run the following vplexcli command to check the current state of the uninterruptable-power-supplies, ll clusters/*/uninterruptible-power-supplies/* Sample Output as reported for cluster-1: VPlexcli:/> ll clusters/*/uninterruptible-power-supplies/* /clusters/cluster-1/uninterruptible-power-supplies/ups-2-a: Name Value ------------------------ ------------- battery-replacement-date 03/25/13 battery-status fully-charged << checked this value for the reported FRU battery-time-remaining 5min operational-status online << checked this value for the reported FRU revision-number 1XE1 serial-number QSxxxxxxxxxx /clusters/cluster-1/uninterruptible-power-supplies/ups-2-b: Name Value ------------------------ ------------- battery-replacement-date 03/24/13 battery-status fully-charged battery-time-remaining 5min operational-status online revision-number 1XE1 serial-number QSxxxxxxxxxx If the battery-status shows as fully-charged and the operational-status shows as online, this issue can be ignored. The alert was most likely a false positive, in that there was a battery test occurring when the FRU monitor portion of the code performed its check of the FRU components.If the battery-status shows charging and the operational-status shows online, check again in about 5 minutes as the battery test may still be in-progress, if after five minutes the check shows the same status contact Dell Customer Support for further assistance.If the battery-status shows charging and the operational-status shows offline or failure this may indicate an issue. Contact Dell Customer Support for further assistance. Standby-Power-Supply (SPS-A or SPS-B) (VS2) Follow the steps to log in as listed above following steps 2 or 3. Run the following vplexcli command to check the state of the Standby-Power-Supplies, ll /engines/*/stand-by-power-supplies/*Sample Output as reported for cluster-1: VPlexcli:/> ll /engines/*/stand-by-power-supplies/* /engines/engine-1-1/stand-by-power-supplies/stand-by-power-supply-a: Attributes: Name Value ---------------------- -------------- battery-status fully-charged << check this value for the reported FRU battery-time-remaining 300 seconds operational-status online << check this value for the reported FRU part-number 078-000-050 revision-number A08 serial-number ACLSPxxxxxxxxx Contexts: Name ------------ conditioning /engines/engine-1-1/stand-by-power-supplies/stand-by-power-supply-b: Attributes: Name Value ---------------------- -------------- battery-status fully-charged battery-time-remaining 300 seconds operational-status online part-number 078-000-049 revision-number A06 serial-number ACxxxxxxxxxxxx If the battery-status shows charging or unknown and the operational-status shows offline or removed, or anything other than what is listed above for these two states, there is a possible issue and you must contact Dell Customer Support for further assistance.If the SPS-states show as listed full-charged, this issue can be ignored. Power-Supplies (ps a or ps b) (VS2 or VS6)Follow steps 2 or 3 to access the VPLEX. Run the following VPlexcli command to check the current state of the power-supplies, ll engines/*/power-supplies/*Sample Output as reported for cluster-1 of a VS2: VPlexcli:/> ll engines/*/power-supplies/* /engines/engine-1-1/power-supplies/power-supply-a0: Name Value ------------------------------ -------------- onDC false << check this value for the reported FRU operational-status online << check this value for the reported FRU part-number 071-000-543 revision-number A08 serial-number ASTACxxxxxxxxx temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-a1: Name Value ------------------------------ -------------- onDC false operational-status online part-number 071-000-543 revision-number A08 serial-number ASTACxxxxxxxxx temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-b0: Name Value ------------------------------ -------------- onDC false operational-status online part-number 071-000-543 revision-number A08 serial-number ASTACxxxxxxxxx temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-b1: Name Value ------------------------------ -------------- onDC false operational-status online part-number 071-000-543 revision-number A08 serial-number ASTACxxxxxxxxx temperature-threshold-exceeded false Sample Output as reported for cluster-1 of a VS6: VPlexcli:/> ll /engines/*/power-supplies/* /engines/engine-1-1/power-supplies/power-supply-a0: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9191501159 temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-a1: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9191501153 temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-b0: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9191403617 temperature-threshold-exceeded false /engines/engine-1-1/power-supplies/power-supply-b1: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9191501130 temperature-threshold-exceeded false /engines/engine-1-2/power-supplies/power-supply-a0: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9185207413 temperature-threshold-exceeded false /engines/engine-1-2/power-supplies/power-supply-a1: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9185207593 temperature-threshold-exceeded false /engines/engine-1-2/power-supplies/power-supply-b0: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9185207960 temperature-threshold-exceeded false /engines/engine-1-2/power-supplies/power-supply-b1: Name Value ------------------------------ -------------- marker-led off onDC false operational-status online part-number 071-000-611-01 revision-number FFF serial-number AC7N9185207592 temperature-threshold-exceeded false If the onDC shows as false and the operational-status shows as online, this issue can be ignored. The alert was most likely a false positive in that there was a battery test occurring when the FRU monitor portion of the code ran its check of the FRU states.If the onDC value shows true and the operational-status shows failure this could indicate an issue. Contact Dell Customer Support for further assistance. Battery-Backup-Unit (BBU-a/b#) - VS6:If you see a call home/alert for the BBUs available on the VS6 only, see article 000167060, "VPLEX: BBUs are frequently cycling from "Charging" to "Fully-Charged" for more details.Related article:000170713 - VPLEX: Event 0x8a4861d6 FRU is not present reported for SPS, Power-Supply, or UPS

Support Cases