Loading...
Loading...
With the CX8 auxiliary cables firmly connected, only around 460 Gb/s bandwidth was observed while running NIC RDMA on the CX8 cards. Auxiliary Cables 18 and 19 are illustrated in the XE8712 figure below: Cables Highlighted in RED rectangle in the M7725 figure below:
The M7725 and XE8712 only support PCIe Gen5. With auxiliary cables connected, the CX8 uses the main x16 slot plus an auxiliary cable in a second x16 slot. Combined, they provide the x32 lanes to achieve 800 Gb/s using Gen5 speeds. By default, the NVIDIA ConnectX-8 operates in PCIe Gen6 x16 mode, which provides the full bandwidth required for 800 Gb/s networking over a single x16 interface.
Download and install MFT from the NVIDIA website: NVIDIA Firmware Tools (MFT) $ mst start $ mst status (this prints the <device name> for the Mellanox adapter, such as, /dev/mst/mt4131_pciconf0) To modify the PCIe width from x16 to x32, perform the following commands: Run the following series of mlxconfig commands to update the PCIe settings: Perform a power cycle on the system to apply the changes. Verify that the CX8 is now operating at x32 by running the following command: $ lspci | grep -i Mellanox (XE8712, Two physical CX8 NICs are presented as four logical interface NICs) $ lspci | grep -i Mellanox (M7725, One physical CX8 NIC are presented as two logical interface NICs) 0000:01:00.0 Mellanox CX8 0000:3f:00.0 Mellanox CX8 Also check iDRAC webGUI > System > Overview > Components > PCIe Slots . Two CX8 devices are listed in the same PCIe slot. Run the RDMA test again, each interface achieves a throughput around 390 Gb/s with total aggregated bandwidth of approximately 780 Gb/s per CX8 card.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.