Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Document Version Release Date Details 5 09/16/2022 Updated the URL for the HPE MLNX-OFED Software Delivery Repository in the Resolution. 4 05/20/2020 Additional update to Scope and Resolution with Linux smart component firmware versions. 3 09/06/2019 Updated Scope and Resolution with Linux smart component firmware versions. 2 10/23/2018 Updated Resolution section with information on functionality added to certain Linux smart component firmware versions. 1 02/13/2018 Original Document Release. When "secure boot" mode is enabled on an HPE ProLiant server, the HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux does not update the firmware on the InfiniBand and Ethernet adapters listed in the Scope section. Below is an example of the message displayed when attempting to update the firmware on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter when the Linux firmware component is run manually from the command line: > ./hpsetup ###################################################################### HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Copyright (c) 2011 Hewlett-Packard Enterprise Development Company,L.P. ###################################################################### List of Network Adapters detected on the Server................. [0] 0000:05:00.0 Intel Corporation [1] 0000:81:00.0 Mellanox Technologies If PSID or FW_Version is not found for some interfaces, please check /tmp/datadSmjj5 Interface 0000:05:00.0 is not Mellanox one. NIC firmware update did not complete. Check log for errors. When Smart Update Manager (SUM) is used to update the network adapter firmware on a server booted in secure boot mode, the SUM inventory process will display the message "Node up to date, No applicable component found" although the baseline includes an applicable firmware smart component.
Any HPE system when "secure boot" mode is enabled attempting to update the network adapter firmware using the following: The following Linux firmware smart components are affected: Manually run from command line or when deployed with Smart Update Manager (SUM) from HPE Service Pack for ProLiant (SPP): firmware-nic-mellanox-ethernet-only RPM (All versions) firmware-hca-mellanox-vpi-connectx4 RPM (All versions) firmware-nic-mellanox-ib-cx4-cx5 RPM (All versions) firmware-hca-mellanox-infiniband-only RPM (All versions) firmware-hca-mellanox-vpi-eth-ib RPM (All versions) Only manually run from command line: firmware-nic-mellanox-nic-mft RPM (All versions) firmware-hca-mellanox-vpi-connectx6-mft (All versions) The above list of affected Linux firmware smart components is current as of the Release Date of this Customer Advisory Revision. Any future smart component will be affected by the issue only when manually run from the command line. The following network adapters are affected when the firmware smart component is manually run from the command line or when deployed with Smart Update Manager (SUM) from HPE Service Pack for ProLiant (SPP): HPE InfiniBand EDR 100Gb 1-port 841QSFP28 Adapter (HPE Part Number: 872725-B21) HPE Apollo InfiniBand EDR 100Gb 2-port 840z Mezzanine FIO Adapter (HPE Part Number: 843400-B21) HPE Ethernet 25Gb 2-port 640SFP28 Adapter (HPE Part Number: 817753-B21) HPE Ethernet 25Gb 2-port 640FLR-SFP28 Adapter (HPE Part Number: 817749-B21) HPE InfiniBand EDR/Ethernet 100Gb 1-port 840QSFP28 Adapter (HPE Part Number: 825110-B21) HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21) HPE Apollo InfiniBand EDR 100Gb 2-port 841z Mezzanine Adapter (HPE Part Number: 872723-B21) HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (HPE Part Number: 879482-B21) HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (HPE Part Number: 872726-B21) HPE Synergy 6410C 25/50Gb Ethernet Adapter (HPE Part Number: 868779-B21) HPE Ethernet 100Gb 1-port 842QSFP28 Adapter (HPE Part Number: 874253-B21) HP Ethernet 10Gb 2-port 546SFP+ Adapter (HPE Part Number: 779793-B21) HP Ethernet 10Gb 2-port 546FLR-SFP+ Adapter (HPE Part Number: 779799-B21) HPE Infiniband FDR 2-port 545QSFP Adapter (HPE Part Number: 702211-B21) HPE Infiniband FDR 2-port 545FLR-QSFP Adapter (HPE Part Number: 702212-B21) HP InfiniBand FDR 2-port 545M Adapter (HPE Part Number: 702213-B21) HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+M Adapter (HPE Part Number: 764282-B21) HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+M Adapter (HPE Part Number: 764283-B21) HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+QSFP Adapter (HPE Part Number: 764284-B21) HPE InfiniBand FDR/Ethernet 10Gb/40Gb 2-port 544+FLR-QSFP Adapter (HPE Part Number: 764285-B21) HPE InfiniBand QDR/Ethernet 10Gb 2-port 544+FLR-QSFP Adapter (HPE Part Number: 764286-B21) The following network adapters are affected only when the firmware smart component is manually run from the command line: HPE Ethernet 10Gb 2-port 548SFP+ Adapter (HPE Part Number: P11338-B21) HPE InfiniBand HDR/Ethernet 200Gb 1-port 940QSFP56 x16 Adapter (HPE Part Number: P06154-B21) HPE InfiniBand HDR100/Ethernet 100Gb 1-port 940QSFP56 x16 Adapter (HPE Part Number: P06250-B21) HPE InfiniBand HDR100/Ethernet 100Gb 2-port 940QSFP56 x16 Adapter (HPE Part Number: P06251-B21) The above list of affected adapters is current as of the Release Date of this Customer Advisory Revision. Any future Mellanox adapter will be affected by the issue only when the firmware smart component is manually run from the command line. Note : Windows and VMware ESXi firmware smart components are not affected by this issue.
The Linux smart components for Mellanox network adapters use a user space firmware tool "mstflint." To access the network adapters firmware listed in the Scope section above, in secure boot mode, a kernel space tool (flint) along with signed kernel module (MST) is needed. As a workaround, use HPE signed "mst" kernel module and "flint" tool from HPE MLNX-OFED Software Delivery Repository to manually update firmware on the network adapters listed in the Scope section above when the server has booted in secure boot mode. The HPE MLNX-OFED Software Delivery Repository is available at the following URL: https://downloads.linux.hpe.com/SDR/project/mlnx_ofed_cx4plus/ A worked example for the firmware upgrade on an HPE InfiniBand EDR/Ethernet 100Gb 2-port 840QSFP28 Adapter (HPE Part Number: 825111-B21) is shown below for reference: To Verify SecureBoot status on the server: [root@localhost ~]# mokutil --sb-state SecureBoot enabled Subscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command: [root@localhost ~]# yum install mft kmod-kernel-mft-mlnx Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 will be installed ---> Package mft.x86_64 0:4.8.0-26 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================== Package Arch Version Repository Size ============================================================================================================================================================== Installing: kmod-kernel-mft-mlnx x86_64 4.8.0-1.rhel7u3 MLNX_OFED 12 k mft x86_64 4.8.0-26 MLNX_OFED 59 M Transaction Summary ============================================================================================================================================================== Install 2 Packages Total download size: 60 M Installed size: 132 M Is this ok [y/d/N]: y Downloading packages: (1/2): kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64.rpm | 12 kB 00:00:01 (2/2): mft-4.8.0-26.x86_64.rpm | 59 MB 00:04:23 -------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 231 kB/s | 60 MB 00:04:23 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 1/2 Installing : mft-4.8.0-26.x86_64 2/2 Verifying : mft-4.8.0-26.x86_64 1/2 Verifying : kmod-kernel-mft-mlnx-4.8.0-1.rhel7u3.x86_64 2/2 Installed: kmod-kernel-mft-mlnx.x86_64 0:4.8.0-1.rhel7u3 mft.x86_64 0:4.8.0-26 Complete! Note: On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx). Start MST modules and identify device name by typing the following command: [root@localhost ~]# service mst start Starting MST (Mellanox Software Tools) driver set Loading MST PCI module - Success Loading MST PCI configuration module - Success Create devices Unloading MST PCI module (unused) - Success [root@localhost ~]# service mst status MST modules: ------------ MST PCI module is not loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt4115_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:81:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 Use the "flint" command to query the current firmware version and PSID of the device: [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q Image type: FS3 FW Version: 12.21.1000 FW Release Date: 29.10.2017 Product Version: rel-12_21_1000 Rom Info: type=UEFI version=14.14.22 cpu=AMD64 type=PXE version=3.5.305 devid=4115 cpu=AMD64 Description: UID GuidsNumber Base GUID: e0071bffff68d0bc 4 Base MAC: 0000e0071b68d0bc 4 Image VSD: N/A Device VSD: N/A PSID: HP_2190110032 Security Attributes: N/A Download latest firmware binary for the adapter from HPE.com Support Center. Query the firmware binary to ensure that PSID matches with the device by typing the following command: [root@localhost ~]# flint -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin q Image type: FS3 FW Version: 12.21.2010 FW Release Date: 27.11.2017 Product Version: rel-12_21_2010 Rom Info: type=UEFI version=14.14.25 cpu=AMD64 type=PXE version=3.5.305 devid=4115 cpu=AMD64 Description: UID GuidsNumber Base GUID: N/A 4 Base MAC: N/A 4 Image VSD: N/A Device VSD: N/A PSID: HP_2190110032 Security Attributes: N/A After PSID is verified, run the following command to update the device firmware: [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 -i fw-ConnectX4-rel-12_21_2010-825111-B21_Ax_Bx-UEFI-14.14.25-FlexBoot-3.5.305.bin burn Current FW version on flash: 12.21.1000 New FW version: 12.21.2010 Burning FW image without signatures - OK Restoring signature - OK -I- To load new FW run mlxfwreset or reboot machine. Reboot the server for firmware update to take effect. After the server is back online, query the device and verify firmware version by typing the following command: [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q Image type: FS3 FW Version: 12.21.2010 FW Release Date: 27.11.2017 Product Version: rel-12_21_2010 Rom Info: type=UEFI version=14.14.25 cpu=AMD64 type=PXE version=3.5.305 devid=4115 cpu=AMD64 Description: UID GuidsNumber Base GUID: e0071bffff68d0bc 4 Base MAC: 0000e0071b68d0bc 4 Image VSD: N/A Device VSD: N/A PSID: HP_2190110032 Security Attributes: N/A Alternatively, starting SPP release 2018.06.0, the Mellanox Linux smart components can be directly run after installing the prerequisite MFT RPMs from SDR to update Mellanox adapter firmware in secure boot mode. This functionality is added to the following Linux smart component firmware versions: firmware-nic-mellanox-ethernet-only-1.0.8-2.1.x86_64.rpm and higher versions firmware-hca-mellanox-vpi-connectx4-1.0.4-1.1.x86_64.rpm and higher versions firmware-nic-mellanox-ib-cx4-cx5-1.0.2-1.1.x86_64.rpm and higher versions firmware-hca-mellanox-infiniband-only-1.0.6-1.1.x86_64.rpm and higher versions firmware-hca-mellanox-vpi-eth-ib-1.0.6-1.1.x86_64.rpm and higher versions A worked example is provided below: To Verify SecureBoot status on the server, type the following command: [root@localhost ~]# mokutil --sb-state SecureBoot enabled Subscribe to MLNX-OFED repository following MLNX-OFED SDR documentation and install the relevant RPMs by typing the following command: [root@localhost ~]# yum install mft kmod-kernel-mft-mlnx Loaded plugins: langpacks, product-id, search-disabled-repos, subscription-manager This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register. Resolving Dependencies --> Running transaction check ---> Package kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 will be installed ---> Package mft.x86_64 0:4.10.0-104 will be installed --> Finished Dependency Resolution Dependencies Resolved ==================================================================== Package Arch Version Repository Size ==================================================================== Installing: kmod-kernel-mft-mlnx x86_64 4.10.0-1.rhel7u5 MLNX_OFED 27 k mft x86_64 4.10.0-104 MLNX_OFED 147 M Transaction Summary ==================================================================== Install 2 Packages Total download size: 147 M Installed size: 147 M Is this ok [y/d/N]: y Downloading packages: (1/2): kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64.rpm |27 kB00:00:01 (2/2): mft-4.10.0-104.x86_64.rpm | 147 MB 00:04:23 -------------------------------------------------------------------- T otal 231 kB/s | 147 MB 00:04:23 Running transaction check Running transaction test Transaction test succeeded Running transaction Warning: RPMDB altered outside of yum. Installing : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 1/2 Installing : mft-4.10.0-104.x86_64 2/2 Verifying : mft-4.10.0-104.x86_64 1/2 Verifying : kmod-kernel-mft-mlnx-4.10.0-1.rhel7u5.x86_64 2/2 Installed: kmod-kernel-mft-mlnx.x86_64 0:4.10.0-1.rhel7u5 mft.x86_64 0:4.10.0-104 Complete! Note : On a server installed with SUSE Linux Enterprise Server, use "zypper install mft kernel-mft-mlnx-kmp-default". The kernel mft RPM name for SLES (kernel-mft-mlnx-kmp-default) is different from that of RHEL (kmod-kernel-mft-mlnx). Install the Linux smart component RPM for the Mellanox adapter and update firmware as shown below: [root@localhost ~]# rpm -ivh firmware-hca-mellanox-vpi-connectx4-1.0.4 - 1.1.x86_64.rpm Preparing... ################################# [100%] Updating / installing... 1:firmware-hca-mellanox-vpi-connect ################################# [100%] [root@localhost ~]# cd /usr/lib/x86_64-linux-gnu/firmware-xxx-* [root@localhost ~]# ./setup ###################################################################### HPE Mellanox InfiniBand Online Firmware Upgrade Utility for Linux Copyright (c) 2011 Hewlett-Packard Enterprise Development Company, L.P. ###################################################################### MST modules: ------------ MST PCI module loaded MST PCI configuration module loaded MST devices: ------------ /dev/mst/mt4115_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:37:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 /dev/mst/mt4119_pciconf0 - PCI configuration cycles access. domain:bus:dev.fn=0000:12:00.0 addr.reg=88 data.reg=92 Chip revision is: 00 Starting MST (Mellanox Software Tools) driver set [warn] mst_pci is already loaded, skipping [warn] mst_pciconf is already loaded, skipping Create devices SecureBoot is enabled. List of Network Adapters detected on the Server................. [0] 0000:02:00.0 Broadcom Limited [1] 0000:12:00.0 Mellanox Technologies [2] 0000:37:00.0 Mellanox Technologies If PSID or FW_Version is not found for some interfaces, please check /tmp/dataEJMPz4 Interface 0000:02:00.0 is not Mellanox one. Mellanox card info for 0000:12:00.0 = FW_VERSION 16.22.1402, BUS_INFO 0000:12:00.0, PSID HPE0000000009 0000:12:00.0 DEVICE INFO---------->15b3 1017 1590 256 pciIdString-------------->15B3-1017-1590-0256-HPE0000000009 Mellanox card info for 0000:37:00.0 = FW_VERSION 12.21.1000, BUS_INFO 0000:37:00.0, PSID HP_2190110032 0000:37:00.0 DEVICE INFO---------->15b3 1013 1590 c8 pciIdString-------------->15B3-1013-1590-00C8-HP_2190110032 Repository has New firmware version for interface 0000:12:00.0.................Please flash the newer version Current Firmware Version is 16.22.1402 on 0000:12:00.0 Repository has these firmware versions............ [1] Image Version 16.22.4030 [2] Image Version 16.22.4030 Would you like to flash the firmware? y/n/q (y):y Current FW version on flash: 16.22.1402 New FW version: 16.22.4030 Initializing image partition - OK Writing Boot image component - OK -I- To load new FW run mlxfwreset or reboot machine. Firmware Flashed: SUCCESS for interface 0000:12:00.0 Repository has New firmware version for interface 0000:37:00.0.................Please flash the newer version Current Firmware Version is 12.21.1000 on 0000:37:00.0 Repository has these firmware versions............ [1] Image Version 12.22.4030 [2] Image Version 12.22.4030 Would you like to flash the firmware? y/n/q (y):y Current FW version on flash: 12.21.1000 New FW version: 12.22.4030 Burning FW image without signatures - OK Restoring signature - OK -I- To load new FW run mlxfwreset or reboot machine. Firmware Flashed: SUCCESS for interface 0000:37:00.0 Please Reboot node for new image to be loaded into silicon. Reboot the server for firmware update to take effect. After the server is back online, query the device and verify firmware version by typing the following command: [root@localhost ~]# flint -d /dev/mst/mt4115_pciconf0 q Image type: FS3 FW Version: 12.22.4030 FW Release Date: 2.4.2018 Product Version: rel-12_22_4030 Rom Info: type=UEFI version=14.15.20 cpu=AMD64 type=PXE version=3.5.404 cpu=AMD64 Description: UID GuidsNumber Base GUID: 98f2b3ffffcc8d54 4 Base MAC: 98f2b3cc8d54 4 Image VSD: N/A Device VSD: N/A PSID: HP_2190110032 Security Attributes: N/A [root@localhost ~] # flint -d /dev/mst/mt4119_pciconf0 q Image type: FS4 FW Version: 16.22.4030 FW Release Date: 2.4.2018 Product Version: 16.22.4030 Rom Info: type=UEFI version=14.15.20 cpu=AMD64 type=PXE version=3.5.404 cpu=AMD64 Description: UID GuidsNumber Base GUID: 040973ffffc91e78 8 Base MAC: 040973c91e78 8 Image VSD: N/A Device VSD: N/A PSID: HPE0000000009 Security Attributes: secure-fw Note : In secure boot mode: Starting SPP release 2019.12.0, SUM offline and online deployment of Mellanox firmware smart components, ensures the "mft" and "kmod-kernel-mft-mlnx" RPMs are installed prior to updating Mellanox adapter firmware in secure boot mode. This functionality is added to the following Linux smart component firmware versions: firmware-nic-mellanox-eth-only-mft-1.0.0-1.1.x86_64.rpm (or later) firmware-hca-mellanox-vpi-connectx6-mft-1.0.1-1.1.rpm (or later) This functionality will be added to any future Linux firmware smart component for Mellanox network adapter. RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL: HPE Email Preference Center NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches and other support software downloads, refer to the Navigation Tips document. SEARCH TIP: For hints on locating similar documents on HPE.com, refer to the Search Tips document.