OPERATIONAL DEFECT DATABASE
...
...
Document VersionRelease DateDetails509/16/2022Updated the URL for the HPE MLNX-OFED Software Delivery Repository in the Resolution.405/20/2020Additional update to Scope and Resolution with Linux smart component firmware versions.309/06/2019Updated Scope and Resolution with Linux smart component firmware versions.210/23/2018Updated Resolution section with information on functionality added to certain Linux smart component firmware versions.102/13/2018Original Document Release.When "secure boot" mode is enabled on an HPE ProLiant server, the HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux does not update the firmware on the InfiniBand and Ethernet adapters listed in the Scope section.Below is an example of the message displayed when attempting to update the firmware on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter when the Linux firmware component is run manually from the command line:> ./hpsetup######################################################################HPE Mellanox InfiniBand Online Firmware Upgrade Utility for LinuxCopyright (c) 2011 Hewlett-Packard Enterprise Development Company,L.P.######################################################################List of Network Adapters detected on the Server.................[0] 0000:05:00.0 Intel Corporation[1] 0000:81:00.0 Mellanox TechnologiesIf PSID or FW_Version is not found for some interfaces, pleasecheck /tmp/datadSmjj5Interface 0000:05:00.0 is not Mellanox one.NIC firmware update did not complete. Check log for errors.When Smart Update Manager (SUM) is used to update the network adapter firmware on a server booted in secure boot mode, the SUM inventory process will display the message "Node up to date, No applicable component found" although the baseline includes an applicable firmware smart component.
Any HPE system when "secure boot" mode is enabled attempting to update the network adapter firmware using the following: The following Linux firmware smart components are affected:Manually run from command line or when deployed with Smart Update Manager (SUM) from HPE Service Pack for ProLiant (SPP):firmware-nic-mellanox-ethernet-only RPM (All versions)firmware-hca-mellanox-vpi-connectx4 RPM (All versions)firmware-nic-mellanox-ib-cx4-cx5 RPM (All versions)firmware-hca-mellanox-infiniband-only RPM (All versions)firmware-hca-mellanox-vpi-eth-ib RPM (All versions)Only manually run from command line:firmware-nic-mellanox-nic-mft RPM (All versions)firmware-hca-mellanox-vpi-connectx6-mft (All versions)The above list of affected Linux firmware smart components is current as of the Release Date of this Customer Advisory Revision. Any future smart component will be affected by the issue only when manually run from the command line.The following network adapters are affected when the firmware smart component is manually run from the command line or when deployed with Smart Update Manager (SUM) from HPE Service Pack for ProLiant (SPP):HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter (HPE Part Number: 872725-B21)HPE Apollo InfiniBand EDR 100Gb 2-port 840z Mezzanine FIO Adapter (HPE Part Number: 843400-B21)HPE Ethernet 25Gb 2-port 640SFP28 Adapter (HPE Part Number: 817753-B21)HPE Ethernet 25Gb 2-port 640FLR-SFP28 Adapter (HPE Part Number: 817749-B21)HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter (HPE Part Number: 825110-B21)HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21)HPE Apollo InfiniBand EDR 100Gb 2-port 841z Mezzanine Adapter (HPE Part Number: 872723-B21)HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (HPE Part Number: 879482-B21)HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (HPE Part Number: 872726-B21)HPE Synergy 6410C 25/50Gb Ethernet Adapter (HPE Part Number: 868779-B21)HPE Ethernet 100Gb 1-port 842QSFP28 Adapter (HPE Part Number: 874253-B21)HP Ethernet 10Gb 2-port 546SFP+ Adapter (HPE Part Number: 779793-B21)HP Ethernet 10Gb 2-port 546FLR-SFP+ Adapter (HPE Part Number: 779799-B21)HPE Infiniband FDR 2-port 545QSFP Adapter (HPE Part Number: 702211-B21)HPE Infiniband FDR 2-port 545FLR-QSFP Adapter (HPE Part Number: 702212-B21)HP InfiniBand FDR 2-port 545M Adapter (HPE Part Number: 702213-B21)HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter (HPE Part Number: 764282-B21)HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter (HPE Part Number: 764283-B21)HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter (HPE Part Number: 764284-B21)HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter (HPE Part Number: 764285-B21)HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter (HPE Part Number: 764286-B21)The following network adapters are affected only when the firmware smart component is manually run from the command line:HPE Ethernet 10Gb 2-port 548SFP+ Adapter (HPE Part Number: P11338-B21)HPE InfiniBand HDR/Ethernet 200Gb 1-port 940QSFP56 x16 Adapter (HPE Part Number: P06154-B21)HPE InfiniBand HDR100/Ethernet 100Gb 1-port 940QSFP56 x16 Adapter (HPE Part Number: P06250-B21)HPE InfiniBand HDR100/Ethernet 100Gb 2-port 940QSFP56 x16 Adapter (HPE Part Number: P06251-B21)The above list of affected adapters is current as of the Release Date of this Customer Advisory Revision. Any future Mellanox adapter will be affected by the issue only when the firmware smart component is manually run from the command line.Note: Windows and VMware ESXi firmware smart components are not affected by this issue.
The Linux smart components for Mellanox network adapters use a user space firmware tool "mstflint." To access the network adapters firmware listed in the Scope section above, in secure boot mode, a kernel space tool (flint) along with signed kernel module (MST) is needed.As a workaround, use HPE signed "mst" kernel module and "flint" tool from HPE MLNX-OFED Software Delivery Repository to manually update firmware on the network adapters listed in the Scope section above when the server has booted in secure boot mode. The HPE MLNX-OFED Software Delivery Repository is available at the following URL:https://downloads.linux.hpe.com/SDR/project/mlnx_ofed_cx4plus/A worked example for the firmware upgrade on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21) is shown below for reference:To Verify SecureBoot status on the server:[root@localhost ~]# mokutil --sb-stateSecureBoot enabledSubscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command:[root@localhost ~]# yum install mft kmod-kernel-mft-mlnxLoaded plugins: langpacks, product-id, search-disabled-repos, subscription-managerThis system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.Resolving Dependencies--> Running transaction check---> Package kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 will be installed---> Package mft.x86_64 0:4.8.0-26 will be installed--> Finished Dependency ResolutionDependencies Resolved==============================================================================================================================================================Package Arch Version Repository Size==============================================================================================================================================================Installing:kmod-kernel-mft-mlnx x86_64 4.8.0-1.rhel7u3 MLNX_OFED 12 kmft x86_64 4.8.0-26 MLNX_OFED 59 MTransaction Summary==============================================================================================================================================================Install 2 PackagesTotal download size: 60 MInstalled size: 132 MIs this ok [y/d/N]: yDownloading packages:(1/2): kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64.rpm | 12 kB 00:00:01(2/2): mft-4.8.0-26.x86_64.rpm | 59 MB 00:04:23--------------------------------------------------------------------------------------------------------------------------------------------------------------Total 231 kB/s | 60 MB 00:04:23Running transaction checkRunning transaction testTransaction test succeededRunning transactionWarning: RPMDB altered outside of yum.Installing : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 1/2Installing : mft-4.8.0-26.x86_64 2/2Verifying : mft-4.8.0-26.x86_64 1/2Verifying : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 2/2Installed:kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 mft.x86_64 0:4.8.0-26Complete!Note:On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx).Start MST modules and identify device name by typing the following command:[root@localhost ~]# service mst startStarting MST (Mellanox Software Tools) driver setLoading MST PCI module - SuccessLoading MST PCI configuration module - SuccessCreate devicesUnloading MST PCI module (unused) - Success[root@localhost ~]# service mst statusMST modules:------------MST PCI module is not loadedMST PCI configuration module loadedMST devices:------------/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.domain:bus:dev.fn=0000:81:00.0 addr.reg=88 data.reg=92Chip revision is: 00Use the "flint" command to query the current firmware version and PSID of the device:[root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 qImage type: FS3FW Version: 12.21.1000FW Release Date: 29.10.2017Product Version: rel-12_21_1000Rom Info: type=UEFI version=14.14.22 cpu=AMD64type=PXE version=3.5.305 devid=4115 cpu=AMD64Description: UID GuidsNumberBase GUID: e0071bffff68d0bc 4Base MAC: 0000e0071b68d0bc 4Image VSD: N/ADevice VSD: N/APSID: HP_2190110032Security Attributes: N/ADownload latest firmware binary for the adapter from HPE.com Support Center.Query the firmware binary to ensure that PSID matches with the device by typing the following command:[root@localhost ~]# flint -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin qImage type: FS3FW Version: 12.21.2010FW Release Date: 27.11.2017Product Version: rel-12_21_2010Rom Info: type=UEFI version=14.14.25 cpu=AMD64type=PXE version=3.5.305 devid=4115 cpu=AMD64Description: UID GuidsNumberBase GUID: N/A 4Base MAC: N/A 4Image VSD: N/ADevice VSD: N/APSID: HP_2190110032Security Attributes: N/AAfter PSID is verified, run the following command to update the device firmware:[root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin burnCurrent FW version on flash: 12.21.1000New FW version: 12.21.2010Burning FW image without signatures - OKRestoring signature - OK-I- To load new FW run mlxfwreset or reboot machine.Reboot the server for firmware update to take effect.After the server is back online, query the device and verify firmware version by typing the following command:[root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 qImage type: FS3FW Version: 12.21.2010FW Release Date: 27.11.2017Product Version: rel-12_21_2010Rom Info: type=UEFI version=14.14.25 cpu=AMD64type=PXE version=3.5.305 devid=4115 cpu=AMD64Description: UID GuidsNumberBase GUID: e0071bffff68d0bc 4Base MAC: 0000e0071b68d0bc 4Image VSD: N/ADevice VSD: N/APSID: HP_2190110032Security Attributes: N/AAlternatively, starting SPP release 2018.06.0, the Mellanox Linux smart components can be directly run after installing the prerequisite MFT RPMs from SDR to update Mellanox adapter firmware in secure boot mode. This functionality is added to the following Linux smart component firmware versions:firmware-nic-mellanox-ethernet-only-1.0.8-2.1.x86_64.rpm and higher versionsfirmware-hca-mellanox-vpi-connectx4-1.0.4-1.1.x86_64.rpm and higher versionsfirmware-nic-mellanox-ib-cx4-cx5-1.0.2-1.1.x86_64.rpm and higher versionsfirmware-hca-mellanox-infiniband-only-1.0.6-1.1.x86_64.rpm and higher versionsfirmware-hca-mellanox-vpi-eth-ib-1.0.6-1.1.x86_64.rpm and higher versionsA worked example is provided below:To Verify SecureBoot status on the server, type the following command:[root@localhost ~]# mokutil --sb-stateSecureBoot enabledSubscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command:[root@localhost ~]# yum install mft kmod-kernel-mft-mlnxLoaded plugins: langpacks, product-id, search-disabled-repos,subscription-managerThis system is not registered to Red Hat Subscription Management. Youcan use subscription-manager to register.Resolving Dependencies--> Running transaction check---> Package kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 will beinstalled---> Package mft.x86_64 0:4.10.0-104 will be installed--> Finished Dependency ResolutionDependencies Resolved====================================================================Package Arch Version Repository Size====================================================================Installing:kmod-kernel-mft-mlnx x86_64 4.10.0-1.rhel7u5 MLNX_OFED 27 kmft x86_64 4.10.0-104 MLNX_OFED 147 MTransaction Summary====================================================================Install 2 PackagesTotal download size: 147 MInstalled size: 147 MIs this ok [y/d/N]: yDownloading packages:(1/2): kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64.rpm |27 kB00:00:01(2/2): mft-4.10.0-104.x86_64.rpm | 147 MB 00:04:23--------------------------------------------------------------------Total 231 kB/s | 147 MB 00:04:23Running transaction checkRunning transaction testTransaction test succeededRunning transactionWarning: RPMDB altered outside of yum.Installing : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 1/2Installing : mft-4.10.0-104.x86_64 2/2Verifying : mft-4.10.0-104.x86_64 1/2Verifying : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 2/2Installed:kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 mft.x86_64 0:4.10.0-104Complete!Note: On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx).Install the Linux smart component RPM for the Mellanox adapter and update firmware as shown below:[root@localhost ~]# rpm -ivh firmware-hca-mellanox-vpi-connectx4-1.0.4-1.1.x86_64.rpmPreparing... ################################# [100%]Updating / installing...1:firmware-hca-mellanox-vpi-connect################################# [100%][root@localhost ~]# cd /usr/lib/x86_64-linux-gnu/firmware-xxx-*[root@localhost ~]# ./setup######################################################################HPE Mellanox InfiniBand Online Firmware Upgrade Utility for LinuxCopyright (c) 2011 Hewlett-Packard Enterprise Development Company, L.P.######################################################################MST modules:------------MST PCI module loadedMST PCI configuration module loadedMST devices: ------------/dev/mst/mt4115_pciconf0 - PCI configuration cycles access.domain:bus:dev.fn=0000:37:00.0 addr.reg=88 data.reg=92 Chip revisionis: 00/dev/mst/mt4119_pciconf0 - PCI configuration cycles access.domain:bus:dev.fn=0000:12:00.0 addr.reg=88 data.reg=92 Chip revisionis: 00Starting MST (Mellanox Software Tools) driver set [warn] mst_pci isalready loaded, skipping [warn] mst_pciconf is already loaded, skippingCreate devicesSecureBoot is enabled.List of Network Adapters detected on the Server.................[0] 0000:02:00.0 Broadcom Limited[1] 0000:12:00.0 Mellanox Technologies[2] 0000:37:00.0 Mellanox TechnologiesIf PSID or FW_Version is not found for some interfaces, pleasecheck /tmp/dataEJMPz4Interface 0000:02:00.0 is not Mellanox one.Mellanox card info for 0000:12:00.0 = FW_VERSION 16.22.1402, BUS_INFO0000:12:00.0, PSID HPE0000000009 0000:12:00.0DEVICE INFO---------->15b3 1017 1590 256pciIdString-------------->15B3-1017-1590-0256-HPE0000000009Mellanox card info for 0000:37:00.0 = FW_VERSION 12.21.1000, BUS_INFO0000:37:00.0, PSID HP_2190110032 0000:37:00.0DEVICE INFO---------->15b3 1013 1590 c8pciIdString-------------->15B3-1013-1590-00C8-HP_2190110032Repository has New firmware version for interface0000:12:00.0.................Please flash the newer versionCurrent Firmware Version is 16.22.1402 on 0000:12:00.0Repository has these firmware versions............[1] Image Version 16.22.4030[2] Image Version 16.22.4030Would you like to flash the firmware?y/n/q (y):yCurrent FW version on flash: 16.22.1402New FW version: 16.22.4030Initializing image partition - OKWriting Boot image component - OK-I- To load new FW run mlxfwreset or reboot machine.Firmware Flashed: SUCCESS for interface 0000:12:00.0Repository has New firmware version for interface0000:37:00.0.................Please flash the newer versionCurrent Firmware Version is 12.21.1000 on 0000:37:00.0Repository has these firmware versions............[1] Image Version 12.22.4030[2] Image Version 12.22.4030Would you like to flash the firmware?y/n/q (y):yCurrent FW version on flash: 12.21.1000 New FW version: 12.22.4030Burning FW image without signatures - OKRestoring signature - OK-I- To load new FW run mlxfwreset or reboot machine.Firmware Flashed: SUCCESS for interface 0000:37:00.0Please Reboot node for new image to be loaded into silicon.Reboot the server for firmware update to take effect.After the server is back online, query the device and verify firmware version by typing the following command:[root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 qImage type: FS3FW Version: 12.22.4030FW Release Date: 2.4.2018Product Version: rel-12_22_4030Rom Info: type=UEFI version=14.15.20 cpu=AMD64type=PXE version=3.5.404 cpu=AMD64Description: UID GuidsNumberBase GUID: 98f2b3ffffcc8d54 4Base MAC: 98f2b3cc8d54 4Image VSD: N/ADevice VSD: N/A PSID: HP_2190110032Security Attributes: N/A[root@localhost ~] # flint -d /dev/mst/mt4119_pciconf0 qImage type: FS4FW Version: 16.22.4030FW Release Date: 2.4.2018Product Version: 16.22.4030Rom Info: type=UEFI version=14.15.20 cpu=AMD64type=PXE version=3.5.404 cpu=AMD64Description: UID GuidsNumberBase GUID: 040973ffffc91e78 8Base MAC: 040973c91e78 8Image VSD: N/ADevice VSD: N/A PSID: HPE0000000009Security Attributes: secure-fwNote: In secure boot mode:Starting SPP release 2019.12.0, SUM offline and online deployment of Mellanox firmware smart components, ensures the "mft" and "kmod-kernel-mft-mlnx" RPMs are installed prior to updating Mellanox adapter firmware in secure boot mode. This functionality is added to the following Linux smart component firmware versions:firmware-nic-mellanox-eth-only-mft-1.0.0-1.1.x86_64.rpm (or later)firmware-hca-mellanox-vpi-connectx6-mft-1.0.1-1.1.rpm (or later)This functionality will be added to any future Linux firmware smart component for Mellanox network adapter.RECEIVE PROACTIVE UPDATES: Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL:HPE Email Preference CenterNAVIGATION TIP:For hints on navigating HPE.com to locate the latest drivers, patches and other support software downloads, refer to theNavigation Tips document.SEARCH TIP:For hints on locating similar documents on HPE.com, refer to theSearch Tips document.
Operating Systems Affected:Not Applicable
No external links available for this bug
Click on a version to see all relevant bugs
Hewlett Packard Enterprise Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
