Loading...
Loading...
There is an interoperability issue between Mellanox 10G NIC, Mellanox 40G NIC, and Cisco ACI. If the node is rebooted, the switch is rebooted, or the link is reset, the link may not come back up. From the node, the link appears as "no carrier". mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:00:00:00:00:00 hwaddr 00:00:00:00:00:00 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier
The main cause of the issue is that older versions of Cisco ACI do not support Auto-Negotiation. Mellanox adapters use a low frequency method for autonegotiation, and Cisco does not support this low frequency method. Mellanox is hard coded for speed and duplex, however, autonegotiate functions still exist and still has a chance of not linking the two peers.
Mellanox has found a way to resolve this by delaying the autonegotiation lock from Cisco. The Mellanox KB can be found here: https://support.mellanox.com/s/article/3rd-Party-Switches-Link-Is-Down-Due-to-Auto-Negotiation (login is required). The command used is dfe-tuning-delay, previous PowerScale cases have proven a delay of 3000 ms allows for a good link up between the two devices. The command referenced can be found here from Cisco:https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/command_references/921/config/b_N9K_Config_Commands_921/b_N9K_Config_Commands_921_chapter_0100.html#wp2120983900 NOTE: It has been evident that regardless of the ACI version that the dfe-tuning-delay is still required. Consult Cisco Support for any questions about this configuration.
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.