Loading...
Loading...
Document VersionRelease DateDetails1009/18/2018Updated Resolution section with information on iLO 4 Firmware 2.61.906/28/2018Updated link in Resolution.806/08/2018Updated Resolution, including updated version of the iLO 4 firmware and Intelligent Provisioning considerations.701/23/2018Added online Intelligent Provisioning update to Resolution.608/22/2017Updated the Resolution with specific steps to be completed.505/24/2017Updated the Resolution with additional details on running the script.412/05/2016Updated the Resolution with additional details on running the script.308/08/2016yUpdated the Resolution with details on running the script.207/21/2016Updated the Description with additional errors that are observed and updated Resolution to include Step 2 through Step 4.103/31/2016Original Document Release.On an HPE Gen8-series or HPE ProLiant Gen9-series servers with HPE Integrated Lights-Out 4 (iLO 4), the NAND flash device may not initialize or mount properly, which may cause a variety of symptoms, which are listed below:AHS ErrorsAHS Logs display a blank date when performing the following:"Select a range of the Active Health log in days From: _______ To:_________"AHS file system mount may fail with (I/O Error) or (No Such Device).The HP Active Health System (AHS) Logs are unable to be downloaded. AHS Data is not available due to a filesystem error.The iLO Diagnostic tab will display the following error message: "Embedded media manager failed initialization" or "The AHS file system mount failed with (No such device) or "The AHS file system mount failed with (I/O error)" or "Controller firmware revision 2.09.00 Could not partition embedded media device."Unable to download AHS log and "bb_dl_disabled" is displayed.Embedded Media ErrorsController Firmware Version 2.09.00 may fail to restart.Unable to partition Embedded media device.Embedded media manager may fail initialization.Intelligent Provisioning ErrorsIntelligent Provisioning will not execute when selecting F10.OneView ErrorsRemote Insight/Integrated Lights-Out Self-Test Error 8192.Unable to register this HP OneView instance with iLO: There was a problem with posting a command to the iLO.Unable to register this HP OneView instance with iLO: The iLO initialization was unable to complete.Unable to determine if this server hardware is being managed by another management system. Received an error from iLO <ip address of iLO> with Error: Blob Store is not yet initialized. and Status: 126
Any HPE ProLiant Gen8-series or HPE ProLiant Gen9-series server running iLO 4.
The resolution for this issue may take several steps that need to be completed in the order specified below.OVERVIEWStep 1)Upgrade the iLO 4 firmware to version 2.61Step 2)Perform a NAND formatStep 3)Check the iLO status If the iLO status is normal, then skip to Step 6 If the iLO status is still degraded, continue to Step 4Step 4)Schedule downtime; AC power-cycle and repeat the NAND formatStep 5)Check the iLO status If the iLO status is normal, continue toStep 6 If the iLO status is still degraded, then skip to Step 7Step 6)Perform these final steps if the system board does not need to be replaced: Reboot the server; reinstall IP; and refresh the server in OneView (if server is managed by OneView)Step 7)If steps 1-4 did not resolve the degraded iLO, replace the system board.Note:The 2.61 iLO 4 firmware is a critical update. As such, HPE requires users to update to this version immediately. Install this update to take advantage of significant improvements to the write algorithm for the embedded 4 GB non-volatile flash memory (also known as the NAND). These improvements increase the NAND lifespan.Additional considerations before performing a NAND formatA NAND format can be performed while the server is online in most cases.Exception: For ESXi hosts booting from the Embedded SD Card - it isstrongly recommended to perform the NAND format with the ESXi OSshutdown.This recommendation also applies when updating the iLO 4firmware or resetting the iLO for ESXi hosts booting from the Embedded SDCardA server AC power removal may be required (prior to the NAND format) in order for the NAND format to be successful. This can be accomplished for ML or DL servers by shutting down the server and disconnecting the power cables for a few seconds. For blade servers, an E-fuse can be accomplished by logging into the OA CLI and typing "reset server #" where "#" is the bay number of the blade.An AuxPwrCycle feature was added in iLO 4 firmware version 2.55 so that the equivalent of an AC power removal can be performed remotely on a server.Required steps to perform after a successful NAND formatIf the server is powered on when performing the NAND format, a server reboot is required after a successful NAND format. This reboot repopulates the RIS and RESTful data on the server during the next server POST with iLO 4 firmware version 2.53 or newer installed. If iLO 4 firmware version 2.50 or older is installed, connect to the iLO CLI (using putty) and use the following command "oemHP_clearRESTapistate" before rebooting the server to repopulate the RIS and RESTful data on the server.Reinstall Intelligent Provisioning (IP) to ensure that it is working properly and reported as installed in the iLO 4 GUI "System Information - Firmware Information" page.If using OneView: After the server is rebooted (to repopulate the RIS and RESTful data on the server), perform a server refresh in OneView; any existing errors in OneView will need to be marked as cleared after the refresh.Detailed stepsStep 1.Upgrade the iLO 4 firmware to version 2.61. To download the firmware. The latest version of the iLO 4 firmware is available as follows:Note:For ESXi hosts booting from the Embedded SD Card (Gen8/Gen9)- it is strongly recommended to perform this step with the ESXi OS shutdown.Click the following link:https://support.hpe.com/hpesc/public/homeEnter a product name (e.g., "DL380 Gen9") in the text search field and wait for a list of products to populate. From the products displayed, identify the desired product and click on the Drivers & software icon to the right of the product.From the Drivers & software dropdown menus on the left side of the page:Select the Software Type - (e.g. Firmware)Select the Software Sub Type - (e.g. Network)For further filtering if needed - Select the specific Operating System from the Operating Environment.Select the latest release of Firmware - Lights-Out Management iLO 4 firmware version 2.61 (or later). Note: To ensure the latest version will be downloaded, click on the Revision History tab to check if a new version of the firmware/driver is available.ClickDownload.Step 2.Perform NAND format using one of the methods detailed in the customer advisory: using one of the methods detailed in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent ProvisioningFind the document at:https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-a00048622en_usNote:iLO 4 firmware version 2.44 (or later) is required to format the NAND using the following steps.Step 3.Check the iLO status using the options available in the customer advisory: HPE Integrated Lights-Out 4 (iLO 4) - How to Format the NAND Used to Store AHS logs, OneView Profiles, and Intelligent Provisioning (find the document at the link in Step 2).If the iLO status is normal based on the above criteria, then skip to Step 6 If the iLO status is still degraded, continue to Step 4Step 4.If the iLO status is still degraded, perform the following steps:a) Schedule a maintenance windowb) Shut down the serverc) Perform an E-fuse (server blade) or AC Power Pull (DL / ML series servers) d) Perform the NAND format again (refer to the instructions in Step 2 above)Step 5.Check the iLO status (refer to the instructions in Step 3 above)If the iLO status is normal, continue to Step 6If the iLO status is still degraded, then skip to Step 7Step 6.Perform these final steps after the NAND format is successful:a) Reboot the serverb) Reinstall Intelligent Provisioning (see additional details below)c) If the server is managed by HPE OneView, perform a server refresh to bring the server back under management.Note:Any existing errors in OneView will need to be marked as cleared after the refresh.Step 7.If steps 1-4 did not resolve the degraded iLO, contact HPE support to arrange a system board replacement. Follow these steps to complete the remediation:a) Open an HPE support case to arrange for a replacement system board / arrange a maintenance windowb) During the maintenance window, shutdown the serverc) Unassign the OneView profile (if the server is under OneView management)d) Replace the system boarde) Enter Server Model and Serial Number via RBSUf) Update to iLO 4 firmware 2.61 (or later)g) Check iLO status (refer to Step 3 above)h) Reassign the OneView profile (if the server is under OneView management)Note: Any existing errors in OneView will need to be marked as cleared after the OneView profile is applied.If additional assistance is needed, contact HPE support and reference Advisory a00019495en_us as follows:Click on the following URL to locate the HPE Customer Support phone number in your country:https://h20195.www2.hpe.com/v2/Getdocument.aspx?docname=A00039121ENW.OneView ConsiderationsOneView relies on the NAND to be accessible to perform many operations such as adding a new server or applying a profile. Because of this there are several things that need to be understood when dealing with this issue in an OneView environment.OneView interaction with the NAND - OneView uses a portion of the NAND called the iLO blob store. If the blob store is not accessible, adding a server to OneView or assigning a profile to the server cannot be completed.Impact of an inaccessible NAND when managed by OneView - If the NAND becomes inaccessible after the server was added in OneView, there are several things that can cause an outage:E-fuse - If an E-fuse reset is performed, the server will not be able to be brought back under management until the NAND issue is remediated and this will cause an unexpected outage.Server is removed and reinserted - This would essentially be the same as performing an E-fuse, so the same information in the E-fuse section applies here.Un-assign a server profile - If a server profile is unassigned, the server profile will not be able to be reassigned until the NAND issue is remediated.Migration from Virtual Connect to OneView - If an enclosure is migrated from Virtual Connect and a server has an inaccessible NAND, that server will not be able to be added properly until the NAND issue is remediated. If an in-service migration is attempted on a server with an inaccessible NAND, an unexpected outage will occur. Reference advisoryhttps://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c05384185for more detail.Formatting of the NAND - The NAND format wipes the iLO blob store that is used by OneView. It is important to issue a server refresh after a successful NAND format to recover the blob that OneView uses.Reinstalling Intelligent Provisioning ConsiderationsThere are online and offline options for reinstalling Intelligent ProvisioningIntelligent Provisioning can be reinstalled online if the server is running Windows (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers only) or Linux (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers). The available online Windows packages are older versions of Intelligent Provisioning.Intelligent Provisioning must be reinstalled offline if the server is running VMware (HPE ProLiant Gen8-series servers/HPE ProLiant Gen9-series servers).The Intelligent Provisioning software download links are provided below. Intelligent Provisioning versionsHPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.x" HPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.xNote:Intelligent Provisioning version 3.x is for HPE ProLiant Gen10-series servers only.Offline method considerations:Run the Intelligent Provisioning Restore Media to restore the Intelligent Provisioning data. Instructions to restore Intelligent Provisioning are as follows:Instructions to create a bootable DVD with the IP image are provided on the HPE Intelligent Provisioning recovery media download site under the Installation Instructions tab.Reference:https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_170e0ac3cb6e4439bf313f137a#tab3Note:The DVD can be used multiple times.Please note the Intelligent Provisioning Recovery Media DVD may be remotely mounted using HPE Integrated Lights-Out 4 (iLO 4) Virtual Media functionality, in order to reinstall Intelligent Provisioning. Additional information is available in the HPE iLO 4 User Guide at the following URL regarding how to mount an ISO image (federated or un-federated) and perform basic virtual media operations. Reference Pages 189 and 223-237:HPE iLO 4 User Guide:https://support.hpe.com/hpsc/doc/public/display?docId=emr_na-c03334051-16In addition, it is possible to write a script that utilizes HPE Integrated Lights-Out 4 (iLO 4) to reinstall Intelligent Provisioning on multiple servers. HPE Lights-Out management processors support an advanced scripting interface for group configuration and server actions. Scripts would need to be customized for the specific environment and task. Sample scripts are available for customers to reference at the following URL: HPE Lights-Out XML Scripting Sample for Windows:https://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_459b8adc29c04317ad1d6a6752Intelligent Provisioning software download linksHPE ProLiant Gen8-series servers are only supported with Intelligent Provisioning 1.xHPE ProLiant Gen9-series servers are only supported with Intelligent Provisioning 2.xa.) Linux RPM located at the following URLs: For HPE Gen8-series servers:http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen8/For HPE Gen9-series servers:http://downloads.linux.hpe.com/SDR/repo/ip/rhel/current/x86_64/gen9/firmware-intelligentprovisioning-ip-2.71-1.1.x86_64.rpmb.) The Windows files are located at the following URLs:For HPE ProLiant Gen8-series servers: Intelligent Provisioning for Microsoft Windows 64-bit Operating Systems 1.64.0.0 (2 Mar 2017) cp031302.exehttps://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_01b900175afc4ae68be6c87b39For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Windows x64 2.50.0.0(6 Jan 2017) cp031091.exehttps://support.hpe.com/hpsc/swd/public/detail?swItemId=MTX_b031497eafd94b9ea2156cdef6c) ISO images (e.g. for VMware) on Hewlett Packard Enterprise Support Center located at the following:https://support.hpe.com/hpesc/public/home/driverHome?sp4ts.oid=1008862660For HPE ProLiant Gen8-series servers: Intelligent Provisioning Recovery Media, version 1.70(9 Oct 2017)For HPE ProLiant Gen9-series servers: Intelligent Provisioning for Gen9 Servers, version 2.70(b)(28 Feb 2018)RECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively via e-mail through HPE Subscriber's Choice. Sign up for Subscriber's Choice at the following URL:Proactive Updates Subscription Form.NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches, and other support software downloads for ProLiant servers and Options, refer to theNavigation Tips document.SEARCH TIP: For hints on locating similar documents on HPE.com, refer to theSearch Tips Document.To search for additional advisories related to Linux, use the following search string:+Advisory +ProLiant -"Software and Drivers" +Linux
Operating Systems Affected:Not Applicable
Click on a version to see all relevant bugs
Hewlett Packard Enterprise Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.