Symptom
After L3 link flap(spine upgrade) the IPv6 Adjacency got into inconsistent state:
# show consistency-checker forwarding ipv6 unicast
IPV6 Consistency check : stopped cause(User Stopped) table_id(0x80000001)
Execution time : 12493 ms ()
Inconsistent adjacencies:
2. slot(1), vrf(default), ipaddr(fe80::7abc:1aff:fede:8823), ifindex(Ethernet1/55), Adjacency inconsistent in FIB Hardware.
4. slot(1), vrf(default), ipaddr(fc00:10:224::8), ifindex(Ethernet1/55), Adjacency inconsistent in FIB Hardware.
IPv6 traffic egressing L3 port is affected. Port shut/no shut did not solve the issue.
Conditions
This a low frequency intermittent problem that occurs with bulk IPv6 route change triggers. It is a timing problem when more than one change is being processed in the database at the same time, affecting the same route(s). It occurs only with IPv6 routes, and examples of triggers are link flaps and bulk remove/re-add of SRv6-TE policies. other bulk change triggers could cause the problem, but only these 2 cases have been observed.
The problem can occur in any IPv6 deployment with version 10.2(1) or lower.
Workaround
If the problem is observed with 10.2(1), restart u6rib.
If the problem is observed with 9.3(x) or lower, device reload is required to restore traffic on the affected routes.
Further Problem Description
This is a problem with the IPv6 routing response to various bulk change triggers, and affects all versions 10.2(1) and earlier. It is a problem due to the implementation of message queue management in response to those triggers, when other changes to IPv6 routes may be happening simultaneously.
Instead of using a queue to store the messages to be sent to ufdm, u6rib in 10.2(1) and below uses a set of pointers to the RNH list and attempts to manage the pointers to simulate a separate queue. The pointer manipulation is flawed in the affected versions when other route changes may be taking place, leading to an incomplete or inaccurate list of route changes being propagated to ufdm in response to bulk change triggers.
In version 10.2(2), the implementation of ufdm message management in u6rib was updated to align with the newer and very reliable change list method used in urib.
