Symptom

+++ A Nexus 7000, 7700, or 9000 device may cause packet loss for traffic destined to prefixes learned via a routing protocol due to a mis-programmed adjacency index in line card hardware. +++ This is especially likely to happen if a recent reconvergence in the routing table caused the egress interface for prefixes to shift from one interface to another. +++ For example, consider the below prefix, 2001:db8:100::/44. In the working state, the control plane routing table reports that this interface is learned via Ethernet1/1. N77# show ipv6 route 2001:db8:100::/44 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8:100::/44, ubest/mbest: 1/0 *via 2001:db8::2/128, [20/0], 00:00:47, bgp-65000, external N77# show ipv6 route 2001:db8::2 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8::/64, ubest/mbest: 1/0, attached *via 2001:db8::2, Eth1/1, [250/0], 00:01:07, am N77# show ipv6 adjacency 2001:db8::2 Flags: # - Adjacencies Throttled for Glean G - Adjacencies of vPC peer with G/W bit R - Adjacencies learnt remotely IPv6 Adjacency Table for VRF default Total number of entries: 1 Address Age MAC Address Pref Source Interface 2001:db8::2 00:01:25 242e.9228.2257 50 icmpv6 Ethernet1/1 +++ The Nexus device is also able to ICMPv6 ping a host in this prefix without issue. N77# ping6 2001:db8:100::1 PING6 2001:db8:100::1 (2001:db8:100::1): 56 data bytes 64 bytes from 2001:db8:100::1: icmp_seq=0 time=1.104 ms 64 bytes from 2001:db8:100::1: icmp_seq=1 time=0.746 ms 64 bytes from 2001:db8:100::1: icmp_seq=2 time=0.702 ms 64 bytes from 2001:db8:100::1: icmp_seq=3 time=0.711 ms 64 bytes from 2001:db8:100::1: icmp_seq=4 time=0.692 ms --- 2001:db8:100::1 ping6 statistics --- 5 packets transmitted, 5 packets received, 0.00% packet loss round-trip min/avg/max = 0.692/0.791/1.104 ms +++ Now, consider a scenario where the eBGP peering through Ethernet1/1 has failed, and the 2001:db8:100::/44 prefix should now be learned via iBGP from another device connected to Ethernet1/15. The control plane routing table agrees that this prefix is learned via iBGP through Ethernet1/15. N77# show ipv6 route 2001:db8:100::/44 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8:100::/44, ubest/mbest: 1/0 *via 2001:db8::2/128, [200/0], 00:00:22, bgp-65123, internal N77# show ipv6 route 2001:db8::2 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8::/64, ubest/mbest: 1/0, attached *via fe80::2, Eth1/15, [110/200], 00:00:25, ospfv3-1, intra N77# show ipv6 adjacency fe80::2 Flags: # - Adjacencies Throttled for Glean G - Adjacencies of vPC peer with G/W bit R - Adjacencies learnt remotely IPv6 Adjacency Table for VRF default Total number of entries: 1 Address Age MAC Address Pref Source Interface fe80::2 00:03:55 8c60.4f31.b342 50 icmpv6 Ethernet1/15 +++ However, there is no reachability to the host in this prefix, which was previously working without issue. This is demonstrated with a failing ICMPv6 ping to this host from the Nexus. N77# ping6 2001:db8:100::1 PING6 2001:db8:100::1 (2001:db8:100::1): 56 data bytes ping6: sendto: No route to host Request 0 timed out ping6: sendto: No route to host Request 1 timed out ping6: sendto: No route to host Request 2 timed out ping6: sendto: No route to host Request 3 timed out ping6: sendto: No route to host Request 4 timed out --- 2001:db8:100::1 ping6 statistics --- 5 packets transmitted, 0 packets received, 100.00% packet loss +++ If we look at the hardware forwarding table for this prefix, we can see that both forwarding engines (denoted by "Dev: 0" and "Dev: 1" in the output below) in the M3 line card inserted in slot 1 have an AdjIdx (Adjacency Index) value of 0x90 as well as a DI (Destination Index) value of 0xcad. These are drop indices causing traffic destined for this prefix to be discarded in hardware. N77# show system internal forwarding ipv6 route 2001:db8:100::/44 detail slot 1 ======= Hardware IPv6 FIB entries for table default/base 2001:db8:100::/44, Dev: 0 , Idx: 0x1b893c, Prio: 0x0 , RPF Flags: , DGT: VPN: 3 AdjIdx: 0x90 , LIFB: 0 , LIF: (0x7fff ), DI: 0xcad DMAC: 0000.0000.0000 SMAC: 0000.0000.0000 RitIdx: 0x20002 2001:db8:100::/44, Dev: 1 , Idx: 0x1b893c, Prio: 0x0 , RPF Flags: , DGT: VPN: 3 AdjIdx: 0x90 , LIFB: 0 , LIF: (0x7fff ), DI: 0xcad DMAC: 0000.0000.0000 SMAC: 0000.0000.0000 RitIdx: 0x20002

Conditions

+++ A Nexus 7000, 7700, or 9000 device +++ Device is running an affected NX-OS software release +++ Recent routing table re-convergence occurred in the network that caused the egress interface for a large quantity of prefixes to change from one interface to another. An example of an event that can cause this re-convergence is a link flap.

Workaround

+++ If a prefix is in the broken state, then installing a static route for the RNH (Recursive Next Hop) of the prefix may correct the mis-programming of the adjacency index in hardware. Once the static route is installed, connectivity to the prefix will resume. If the static route is uninstalled, connectivity to the prefix should persist without issue. +++ For example, consider the 2001:db8:100::/44 prefix that was in the broken state as described by the "Symptoms" section above. N77# show ipv6 route 2001:db8:100::/44 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8:100::/44, ubest/mbest: 1/0 *via 2001:db8::2/128, [200/0], 00:00:22, bgp-65123, internal <<< This is our RNH N77# show ipv6 route 2001:db8::2 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8::/64, ubest/mbest: 1/0, attached *via fe80::2, Eth1/15, [110/200], 00:00:25, ospfv3-1, intra +++ To work around this issue, one may configure a static route for the RNH of this prefix with a next-hop IPv6 address of the device that all prefixes should have converged through. In this scenario, the next-hop IPv6 address is 2001:db8:6:5123::2 through Ethernet1/15. N77# configure terminal N77(config)# ipv6 route 2001:db8::2/128 2001:db8:6:5123::2 N77(config)# end +++ We can confirm the static route for the RNH was installed correctly. N77# show ipv6 route 2001:db8::2 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8::2/128, ubest/mbest: 1/0 *via 2001:db8:6:5123::2, Eth1/15, [1/0], 00:00:47, static +++ We can confirm that reachability from the Nexus itself to a host in the 2001:db8:100::/44 prefix is now working as expected. N77# ping6 2001:db8:100::1 PING6 2001:db8:100::1 (2001:db8:100::1): 56 data bytes 64 bytes from 2001:db8:100::1: icmp_seq=0 time=0.814 ms 64 bytes from 2001:db8:100::1: icmp_seq=1 time=0.806 ms 64 bytes from 2001:db8:100::1: icmp_seq=2 time=0.792 ms 64 bytes from 2001:db8:100::1: icmp_seq=3 time=0.745 ms 64 bytes from 2001:db8:100::1: icmp_seq=4 time=0.693 ms +++ We can now remove the previously-configured static route. N77# configure terminal N77(config)# no ipv6 route 2001:db8::2/128 2001:db8:6:5123::2 N77(config)# end +++ The control plane re-converges as expected, and reachability from the Nexus device to the previously-tested host persists. N77# show ipv6 route 2001:db8:100::/44 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8:100::/44, ubest/mbest: 1/0 *via 2001:db8::2/128, [200/0], 00:01:28, bgp-65123, internal N77# show ipv6 route 2001:db8::2 IPv6 Routing Table for VRF "default" '*' denotes best ucast next-hop '**' denotes best mcast next-hop '[x/y]' denotes [preference/metric] 2001:db8::/64, ubest/mbest: 1/0, attached *via fe80::2, Eth1/15, [110/200], 00:00:37, ospfv3-1, intra N77# ping6 2001:db8:100::1 PING6 2001:db8:100::1 (2001:db8:100::1): 56 data bytes 64 bytes from 2001:db8:100::1: icmp_seq=0 time=0.832 ms 64 bytes from 2001:db8:100::1: icmp_seq=1 time=0.814 ms 64 bytes from 2001:db8:100::1: icmp_seq=2 time=0.892 ms 64 bytes from 2001:db8:100::1: icmp_seq=3 time=0.793 ms 64 bytes from 2001:db8:100::1: icmp_seq=4 time=0.803 ms

Further Problem Description