Symptom
Traffic is forwarded over backup path eventhough primary path intact.
Conditions
Issue seen with specific sequence of interface bringup triggering BGP PIC.
* The likely sequence is,
*
* 1 - after upgrading to 7.1.2, unshut many interfaces
* 2 - ISIS coming up, but not fully ready
* 3 - BGP coming up, able to update FIB with primary and backup paths.
* 4 - FRR event, likely BFD down. This is not expected
* -- FIB switched to use backup path, and updated PD for that.
* 5 - ISIS continues its setup, BGP primary path becomes TI-LFA ready
* 6 - ISIS continues its setup. BGP backup path becomes ECMP
* --- fib unbinds FRR object, switched back to primary path
* --- but didn't not update PD.
The sequence is derived based on the information saved in traces. It's not actually observed. It's possible that other sequences exist that could lead to the same end state. But we don't know what these sequences are.
Workaround
*** This workaround only applies to router bootup ***
Disable BGP PIC, before bringing up the interfaces. Re-enable BGP PIC, after IGP becomes stable.
Recovery:
if only one or few prefixes impacted then you can use clear route command
clear route vrf /len
clear route /len
if not sure on how many prefixes might be impacted, reprogramming entire CEF table might be worth considering. This can be done through
1. First showdown all the BGP sessions to RR's and then unshut them.
Further Problem Description
This issue existed since 7.0.2 release. Plus the odd releases of 6.7.x. Confirmed platform is asr9k.
If the issue is hit, and the prefix is known, one may use the following commands to recover
1 - clear route [ipv4|ipv6] unicast
2 - clear route vrf [ipv4|ipv6] unicast
Additional note:
This defect CSCvy42280 may not fix the issue completely. Please evaluate below two defects as well.
- CSCvy99058: PIC FRR NH gets reset after 4-min as BGP update missing
- Not required any releases starting from 7.3.x
- CSCvv14988: UFI:BGP PIC convergence of 4-5 secs for VPNv6 prefix
