Symptoms

There is an interoperability issue between Mellanox 10G NIC, Mellanox 40G NIC, and Cisco ACI. If the node is rebooted, the switch is rebooted, or the link is reset, the link may not come back up. From the node, the link appears as "no carrier". mlxen2: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=ed07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> ether 00:00:00:00:00:00 hwaddr 00:00:00:00:00:00 nd6 options=29<PERFORMNUD,IFDISABLED,AUTO_LINKLOCAL> media: Ethernet autoselect status: no carrier

Cause

The main cause of the issue is that older versions of Cisco ACI do not support Auto-Negotiation. Mellanox adapters use a low frequency method for autonegotiation, and Cisco does not support this low frequency method. Mellanox is hard coded for speed and duplex, however, autonegotiate functions still exist and still has a chance of not linking the two peers.

Resolution

Mellanox has found a way to resolve this by delaying the autonegotiation lock from Cisco. The Mellanox KB can be found here: https://support.mellanox.com/s/article/3rd-Party-Switches-Link-Is-Down-Due-to-Auto-Negotiation (login is required). The command used is dfe-tuning-delay, previous PowerScale cases have proven a delay of 3000 ms allows for a good link up between the two devices. The command referenced can be found here from Cisco:https://www.cisco.com/c/en/us/td/docs/switches/datacenter/nexus9000/sw/9-x/command_references/921/config/b_N9K_Config_Commands_921/b_N9K_Config_Commands_921_chapter_0100.html#wp2120983900 NOTE: It has been evident that regardless of the ACI version that the dfe-tuning-delay is still required. Consult Cisco Support for any questions about this configuration.

Support Cases