Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Document Version Release Date Details 2 04/22/2021 Updated to include the following information: IMPORTANT: Ensure iLO Firmware version 2.31 (or later) is in use to properly detect NVMe SSD drives. In addition, reset iLO prior to running the tool to ensure iLO detects the most current drive information. Reference the iLO5 user guide for details. 1 04/05/2021 Original document release IMPORTANT: This 4ICRHPK3 firmware is considered a critical fix and is required to address the issue detailed below. HPE strongly recommends immediate application of this critical fix in the following order: Run the SSD Issue Detection Tool provided in the Resolution section, save the resulting output, and update the firmware and/or replace the drive based on the tool output. HPE was notified by a Solid State Drive (SSD) manufacturer of a firmware issue affecting certain NVMe SSD models (reference the table below) used in a number of HPE server and storage products. Neglecting to update to SSD Firmware Version 4ICRHPK3 may result in drive failure and data loss after 4700 hours of operation. A drive failure may require restoration of data from backup in non-fault tolerance, such as RAID 0, and in fault tolerance RAID mode if more drives fail than what is supported by the fault tolerance RAID mode logical drive. In addition, a drive that has already attained 5900 hours of operation on firmware version HPK1 must be replaced. By disregarding this notification and not performing the recommended resolution, the customer accepts the risk of incurring future related errors. NOTE: The following platforms are NOT AFFECTED by this issue: HPE 3PAR StoreServ Storage, D8000 Disk Enclosure, Nimble Storage, Primera Storage, StoreOnce Systems, XP Storage, HPE StoreEasy 5000 Storage, MSA Storage, SimpliVity, Cloudline, Moonshot and Edgeline. The issue affects SSDs with an HPE firmware version prior to 4ICRHPK3 that may result in SSD failure at 4700 hours of operation (i.e., 6 to 7 months). After the SSD failure occurs, neither the SSD nor the data can be recovered. IMPORTANT: Drives with HPK0 (4ICRHPK0) firmware are NOT susceptible and do not fail after 4700 hours. The script will not flag HPK0 drives as needing to be replaced. However, the SSD Issue Detection Tool referenced in the Resolution section below will generate a message to upgrade the firmware within 14 days. Customers should update to 4ICRHPK3 in order to ensure that the system contains the latest fixes and enhancements. Note: SSD Drive Firmware versions prior to "4ICRHPK3" were shown on HPE Support Center firmware download pages as HPK0 and HPK1. The firmware version installed on the drive is shown as "4ICRHPK0" or "4ICRHPK1.""4ICRHPK0" is the same firmware as HPK0. "4ICRHPK1" is the same firmware as HPK1. For more details, refer to the drive firmware Release Notes for the version with this fix listed in the Resolution section.
Any of the following HPE NVMe SSD models running a drive firmware version prior to 4ICRHPK3: IMPORTANT NOTE: The list of impacted SSD models identified in this documentation is comprehensive. No other drive models are affected. In addition, drives contained within SAN-attached or other external storage sources must be checked separately from the drives in the main system. HPE Model Number HPE SKU HPE SKU DESCRIPTION MO001600KWVUU P13699-B21, P13699-H21, P13699-K21 HPE 1.6TB NVMe x4 Lanes Mixed Use SSD MO001600KWVUU P18140-B21 HPE DX 1.6TB NVMe x4 Lanes Mixed Use SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD VO002000KWVVC P13695-B21, P13695-H21, P13695-K21 HPE 2TB NVMe x4 Lanes Read Intensive SSD VO002000KWVVC P18145-B21 HPE DX 2TB NVMe x4 Lanes Read Intensive SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD MO003200KWVUV P13701-B21,P13701-H21,P13701-K21 HPE 3.2TB NVMe x4 Lanes Mixed Use SSD MO003200KWVUV P18142-B21 HPE DX 3.2TB NVMe x4 Lanes Mixed Use SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD VO004000KWVUR P13697-B21, P13697-H21, P13697-K21 HPE 4TB NVMe x4 Lanes Read Intensive SSD VO004000KWVUR P18149-B21 HPE DX 4TB NVMe x4 Lanes Read Intensive SFF (2.5in) SCN 3yr Wty Digitally Signed Firmware FIO SSD MO006400KWVVA P13703-B21,P13703-H21, P13703-K21 HPE 6.4TB NVMe x4 Lanes Mixed Use SSD
IMPORTANT: Ensure that the SSD Issue Detection Tool is run, and all output is captured, BEFORE updating to the 4ICRHPK3 Firmware. IMPORTANT: Ensure iLO Firmware version 2.31 (or later) is in use to properly detect NVMe SSD drives. In addition, reset iLO prior to running the tool to ensure iLO detects the most current drive information. Reference the iLO5 user guide for details. Immediately upgrade the drive firmware to version 4ICRHPK3, which HPE has released to prevent the issue described above. In some scenarios, the drive firmware cannot be upgraded and the drive must instead be replaced. In order to determine if a drive can be upgraded with the 4ICRHPK3 firmware, or if the drive must be replaced, HPE has developed an SSD Issue Detection Tool that runs a script. The output of the tool will indicate if the firmware should be updated or if a firmware update is not possible, and instead, the drive must be replaced. NOTE: Drives with HPK0 firmware are NOT susceptible and do not fail after 4700 hours. The script will not flag HPK0 drives as needing to be replaced. However, the SSD Issue Detection Tool will generate a message to upgrade the firmware within 14 days. Customers should update to 4ICRHPK3 in order to ensure that the system contains the latest fixes and enhancements. Reminder: This script queries the HPE NVMe drives through Integrated Lights-Out (iLO) and only detects drives attached to PCIe. The script will use iLOREST to check iLO for HPE NVMe SSD drives that are affected by this Customer Bulletin. Note: iLOREST binaries are required to run these scripts. Ensure the scripts are run from the same location as the included iLOREST binaries. Download the SSD Issue Detection Tool at the link below and refer to the associated Readme file for specific instructions on execution and usage of the tool. IMPORTANT:Ensure that the SSD Issue Detection Tool is run, and all output is captured, BEFORE updating to the 4ICRHPK3 Firmware. IMPORTANT: Ensure that all your data is backed up prior to any updates. IMPORTANT: Ensure iLO Firmware version 2.31 (or later) is in use to properly detect NVMe SSD drives. In addition, reset iLO prior to running the tool to ensure iLO detects the most current drive information. Reference the iLO5 user guide for details. On Windows and Linux Platforms: Every drive identified will have a color-coded action. Red actions need to be done immediately. Yellow actions need to be done within 7 days. Cyan actions need to be done within 14 days. Green actions require no action since the firmware is 4ICRHPK3 or later. The actions will include: replace drive, update to 4ICRHPK3, or no action required. The 4ICRHPK3 firmware update requires a reboot to activate the firmware. On VMware Platforms: The color-coded output may not be available on VMware platforms. For VMware Operating Systems, refer to the text output for the appropriate action. The output will be the same, but without the color-coding. The SSD Issue Detection Tool is available at the following URL: The SSD Issue Detection Tool Note: The signature file for the SSD Issue Detection Tool is available here: SSD Issue Detection Tool Signature File Once the tool has been run, output similar to the following will be displayed: As noted in the script output: In Windows and Linux, a RED color that is returned indicates that the action should be performed immediately. For VMware, refer to the noted action. In Windows and Linux, a YELLOW color that is returned indicates that the system should be updated to the HPK3 firmware within 7 days. For VMware, refer to the noted action. In Windows and Linux, a CYAN color that is returned indicates that the system should be updated to the HPK3 firmware within 14 days. For VMware, refer to the noted action. 4ICRHPK3 Firmware is available as follows: VMware: 4ICRHPK3 Firmware For VMware Operating Systems Windows: 4ICRHPK3 Firmware for Windows Operating Systems Linux: 4ICRHPK3 Firmware for Linux Operating Systems IMPORTANT: Drives MUST be replaced in either of the following scenarios: Scenario 1: Once a drive has exceeded 5900 power-on hours, recovery of the drive is no longer possible. Even if the firmware has been updated to 4ICRHPK3, and the drive appears to be functioning properly, it is not and must be replaced. Scenario 2: The drive has asserted. For drive replacement, contact HPE Support. When contacting HPE Support regarding this issue, reference the following document ID: a00112347. Note: After the drive has been replaced, re-run the tool to determine if the drive has 4ICRHPK3 firmware installed. If it does not, follow the download instructions above to obtain the updated firmware. RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail. Sign up for Support Alerts at the following URL: Proactive Updates Subscription Form. NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches and other support software downloads, refer to the Navigation Tips document. SEARCH TIP: For hints on locating similar documents on HPE.com, refer to the Search Tips document.