Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
On any HPE server configured with any of the HPE Mellanox based network adapters listed in the Scope section below, using an HPE 25Gb SFP28 SR 100m Transceiver; the network adapters may not link after upgrading firmware to certain firmware versions. This issue has been reported only on platforms using certain HPE 25Gb SFP28 SR 100m Transceivers and certain Cisco Nexus 9000 series switches. To identify the affected transceivers, perform the "ethtool -m" command: From ethtool -m ... Vendor name : FINISAR CORP. Vendor OUI : 00:90:65 Vendor PN : FTLF8536P4BCL-HP ...
Any HPE server configured with any of the following HPE Mellanox based network adapters using an HPE 25Gb SFP28 SR 100m Transceiver (845398-B21): HPE Network Adapter Firmware version affected HPE Ethernet 10/25Gb 2-port FLR-SFP28 MCX4121A-ACFT Adapter (817749-B21 / 817749R-B21) 14.29.1016 HPE Ethernet 10/25Gb 2-port SFP28 MCX4121A-ACUT Adapter (817753-B21 / 817753R-B21) 14.29.1016 Mellanox MCX512F-ACHT Ethernet 10/25Gb 2-port SFP28 Adapter for HPE (P13188-B21) 16.29.1016 Mellanox MCX562A-ACAI Ethernet 10/25Gb 2-port SFP28 OCP3 Adapter for HPE (P10112-B21) 16.29.1016 HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (872726-B21 / 872726-H21) 16.29.1016 HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (879482-B21) 16.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe3 x16 MCX653105A-HDAT Adapter (P06154-B21 / P06154-H21) 20.29.1016 HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe3 x16 MCX653105A-ECAT Adapter (P06250-B21 / P06250-H21) 20.29.1016 HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe3 x16 MCX653106A-ECAT Adapter (P06251-B21 / P06251-H21) 20.29.1016 HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe4 x16 MCX653105A-ECAT Adapter (P23665-B21 / P23665-H21) 20.29.1016 HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe4 x16 MCX653106A-ECAT Adapter (P23666-B21 / P23666-H21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 MCX653105A-HDAT Adapter (P23664-B21 / P23664-H21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 MCX653436A-HDAI OCP3 PCIe4 x16 Adapter (P31348-B21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 OCP3 MCX653436A-HDAI Adapter (P31348-H21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 MCX653435A-HDAI OCP3 PCIe4 x16 Adapter (P31323-B21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 OCP3 MCX653435A-HDAI Adapter (P31323-H21) 20.29.1016 HPE Ethernet 100Gb 1-port QSFP28 PCIe3 x16 MCX515A-CCAT Adapter (P31246-B21 / P31246-H21) 20.29.1016 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 MCX653106A-HDAT Adapter (P31324-B21 / P31324-H21) 20.29.1016 HPE Ethernet 10/25Gb 2-port SFP+ MCX4621A-ACAB OCP3 Adapter (P11341-B21) 14.29.1016
To resolve this issue, download and install the following firmware versions: HPE Network Adapter Firmware Version Firmware Download HPE Ethernet 10/25Gb 2-port FLR-SFP28 MCX4121A-ACFT Adapter (817749-B21 / 817749R-B21) HPE Ethernet 10/25Gb 2-port SFP28 MCX4121A-ACUT Adapter (817753-B21 / 817753R-B21) 14.31.1200 VMware ESXI 6.5 VMware ESXi 6.7 VMware ESXi 7.0 Microsoft Windows Linux Mellanox MCX512F-ACHT Ethernet 10/25Gb 2-port SFP28 Adapter for HPE (P13188-B21) 16.31.1200 OS Independent Mellanox MCX562A-ACAI Ethernet 10/25Gb 2-port SFP28 OCP3 Adapter for HPE (P10112-B21) 16.31.1200 OS Independent HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (872726-B21 / 872726-H21) 16.31.1200 VMware ESXi 6.5 VMware ESXi 6.7 VMware ESXi 7.0 Microsoft Windows Linux HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (879482-B21) 16.31.1200 VMware ESXi 6.5 VMware ESXi 6.7 VMware ESXi 7.0 Microsoft Windows Linux HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe3 x16 MCX653105A-HDAT Adapter (P06154-B21 / P06154-H21) NA Perform the workaround below. HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe3 x16 MCX653105A-ECAT Adapter (P06250-B21 / P06250-H21) NA Perform the workaround below. HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe3 x16 MCX653106A-ECAT Adapter (P06251-B21 / P06251-H21) NA Perform the workaround below. HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe4 x16 MCX653105A-ECAT Adapter (P23665-B21 / P23665-H21) NA Perform the workaround below. HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe4 x16 MCX653106A-ECAT Adapter (P23666-B21 / P23666-H21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 MCX653105A-HDAT Adapter (P23664-B21 / P23664-H21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 MCX653436A-HDAI OCP3 PCIe4 x16 Adapter (P31348-B21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 OCP3 MCX653436A-HDAI Adapter (P31348-H21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 MCX653435A-HDAI OCP3 PCIe4 x16 Adapter (P31323-B21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 OCP3 MCX653435A-HDAI Adapter (P31323-H21) NA Perform the workaround below. HPE Ethernet 100Gb 1-port QSFP28 PCIe3 x16 MCX515A-CCAT Adapter (P31246-B21 / P31246-H21) NA Perform the workaround below. HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 MCX653106A-HDAT Adapter (P31324-B21 / P31324-H21) NA Perform the workaround below. HPE Ethernet 10/25Gb 2-port SFP+ MCX4621A-ACAB OCP3 Adapter (P11341-B21) 14.31.1200 VMware ESXi 6.5 VMware ESXi 6.7 VMware ESXi 7.0 Microsoft Windows Linux Workaround To avoid this issue on systems where the firmware versions above cannot be installed or there is no firmware version that contains a fix, perform either of the following actions: Downgrade the network adapter firmware to following firmware versions: HPE Network Adapter Firmware version HPE Ethernet 10/25Gb 2-port FLR-SFP28 MCX4121A-ACFT Adapter (817749-B21 / 817749R-B21) 14.27.4000 HPE Ethernet 10/25Gb 2-port SFP28 MCX4121A-ACUT Adapter (817753-B21 / 817753R-B21) 14.27.4000 Mellanox MCX512F-ACHT Ethernet 10/25Gb 2-port SFP28 Adapter for HPE (P13188-B21) 16.27.2008 Mellanox MCX562A-ACAI Ethernet 10/25Gb 2-port SFP28 OCP3 Adapter for HPE (P10112-B21) 16.27.2008 HPE InfiniBand EDR/Ethernet 100Gb 2-port 841QSFP28 Adapter (872726-B21 / 872726-H21) 16.27.2008 HPE InfiniBand FDR/Ethernet 40/50Gb 2-port 547FLR-QSFP Adapter (879482-B21) 16.27.2008 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe3 x16 MCX653105A-HDAT Adapter (P06154-B21 / P06154-H21) 20.27.6008 HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe3 x16 MCX653105A-ECAT Adapter (P06250-B21 / P06250-H21) 20.27.6008 HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe3 x16 MCX653106A-ECAT Adapter (P06251-B21 / P06251-H21) 20.27.6008 HPE InfiniBand HDR100/Ethernet 100Gb 1-port QSFP56 PCIe4 x16 MCX653105A-ECAT Adapter (P23665-B21 / P23665-H21) 20.27.6008 HPE InfiniBand HDR100/Ethernet 100Gb 2-port QSFP56 PCIe4 x16 MCX653106A-ECAT Adapter (P23666-B21 / P23666-H21) 20.27.6008 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 MCX653105A-HDAT Adapter (P23664-B21 / P23664-H21) 20.27.6008 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 MCX653436A-HDAI OCP3 PCIe4 x16 Adapter (P31348-B21) 20.28.4000 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 OCP3 MCX653436A-HDAI Adapter (P31348-H21) 20.28.4000 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 MCX653435A-HDAI OCP3 PCIe4 x16 Adapter (P31323-B21) 20.28.4000 HPE InfiniBand HDR/Ethernet 200Gb 1-port QSFP56 PCIe4 x16 OCP3 MCX653435A-HDAI Adapter (P31323-H21) 20.28.4000 HPE Ethernet 100Gb 1-port QSFP28 PCIe3 x16 MCX515A-CCAT Adapter (P31246-B21 / P31246-H21) 16.28.4000 HPE InfiniBand HDR/Ethernet 200Gb 2-port QSFP56 PCIe4 x16 MCX653106A-HDAT Adapter (P31324-B21 / P31324-H21) 20.28.4000 HPE Ethernet 10/25Gb 2-port SFP+ MCX4621A-ACAB OCP3 Adapter (P11341-B21) 14.27.4000 OR Disable the Forward Error Correction (FEC) and force speed to 25G on the network adapter and the switch port. Consult the switch users guide for instructions on how to disable Forward Error Correction (FEC) on the switch port. To disable the Forward Error Correction (FEC) on the network adapter perform the following steps: Download and install the mlxlink tool available at the following URL: Mellanox Firmware Tools (MFT) Start the MST service and identify <DEVICE> : # mst start # mst status Note: The name of device shown in the outputshould be something like /dev/mst/mt4119_pciconf0 (ConnectX 5) and will refer to it as DEVICE. Run the mlxlink command: # mlxlink -d <DEVICE> -m > mlxlink.out Verify that the system is running the affected network adapter firmware versions listed in the Scope section above and verify if the link is down. On the server affected, set the link mode to 25G with no Forward Error Correction (FEC) by performing the following steps: // Force 25G and FEC NONE (-k NF) # mlxlink -d <DEVICE> -k NF --fec_speed 25G // Toggle link (down/UP). This is IMPORTANT # mlxlink -d <DEVICE> -a TG Set the Switch port to 25G with no Forward Error Correction (FEC). Notes: The commands for Microsoft Windows are the same in the powershell, the device name under "mst status" will be slightly different (make sure to run the shell as admin). For VMware ESXi, add the Mellanox Firmware Tools (MFT) to the default system path or run them using the full path with the command: Example: /opt/mellanox/bin/mlxlink -d <DEVICE> -k NF --fec_speed 25G To export the system path where the mlxlink tool can be run, perform this command: export PATH=$PATH:/opt/mellanox/bin For more information, refer to the following URL: How-to: Install Mellanox Firmware Tools (MFT) on VMware ESXi 6.5 and above. Disclaimer: One or more of the links above will take you outside the HPE website. HPE is not responsible for content outside of its domain. RECEIVE PROACTIVE UPDATES : Receive support alerts (such as Customer Advisories), as well as updates on drivers, software, firmware, and customer replaceable components, proactively in your e-mail through HPE Support Alerts. Sign up for Support Alerts at the following URL: Proactive Updates Subscription Form. NAVIGATION TIP: For hints on navigating HPE.com to locate the latest drivers, patches and other support software downloads, refer to the Navigation Tips document. SEARCH TIP: For hints on locating similar documents on HPE.com, refer to the Search Tips document.