Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
L3VPN with BGP PIC: Traffic loss since it is being sent to remote backup PE incorrectly. it is happening on BGP-PIC scenario, the key point is: LWLDI chain hook to stale ECMP-FEC entry (backup path entry), so the packet will be forward to remote backup path.
Seen on 7721 when the deployment of BGP PIC was with PE node redundancy.
Removal of bgp pic is the workaround, if acceptable
When the primary and backup paths for VRF routes in the case of L3VPN are learnt on the same interface and that particular interface is flapped, traffic is moved to the backup path. However, after four minutes when the FRR reset happened, then new FRRNH object was created, we can see that as part of FRRNH creation, now new IFEC is created. RSHLDI Modify triggered after FRRNH was created, and RSHLDI destination FEC is correctly updated with new IFEC, but there is no LWLDI modification happened, that is the reason LWLDI is still pointing to old IFEC. Example: (enode-b)-------TenGigE0/4/0/1.2310--->(A2H-C0028-01)-------->----------MPLS L3vpn------------>------- (primary BGP PIC PE 10.151.3.33) ----------- MME. 10.150.160.147 | |------------------>--------(backup BGP PIC PE 10.151.3.25) ------------ RP/0/RP0/CPU0:NCS560#show cef vrf VRF_1 10.150.160.147 hardware egress detail loc 0/rp0/cpu0 Tue Oct 17 11:29:30.508 EDT 10.150.160.147/32, version 2307215, internal 0x5000001 0x30 (ptr 0x92d35d48) [1], 0x600 (0x97d4cc78), 0xa08 (0x990f2e50) Updated Oct 13 01:10:10.322 Prefix Len 32, traffic index 0, precedence n/a, priority 3 gateway array (0x92b3bbb0) reference count 4, flags 0x100038, source rib (7), 0 backups [5 type 1 flags 0x441 (0x988b5bb8) ext 0x0 (0x0)] LW-LDI[type=1, refc=1, ptr=0x97d4cc78, sh-ldi=0x988b5bb8] gateway array update type-time 3 Sep 28 02:05:52.938 LDI Update time Sep 28 04:10:30.033 LW-LDI-TS Sep 28 02:05:52.938 via 10.151.3.25/32, 11 dependencies, recursive, backup [flags 0x6100]. <<<<<<<<<<<<< 10.151.3.25 is the remote edge BGP-PIC backup PE, IGP output labels is 27388 path-idx 0 NHID 0x0 [0x99c24958 0x0] recursion-via-/32 next hop VRF - 'default', table - 0xe0000000 next hop 10.151.3.25/32 via 27398/0/21 next hop 10.218.149.78/32 Te0/5/0/5 labels imposed {27388 27307} via 10.151.3.33/32, 10 dependencies, recursive [flags 0x6000]. <<<<<<<<<<<<< 10.151.3.33 is the remote edge BGP-PIC primary PE. IGP output labels is 27396 path-idx 1 NHID 0x0 [0x9d022098 0x0], Internal 0x9bc540d0 recursion-via-/32 next hop VRF - 'default', table - 0xe0000000 next hop 10.151.3.33/32 via 27406/0/21 next hop 10.218.149.78/32 Te0/5/0/5 labels imposed {27396 27452} Show-data Print at RPLC LEAF - HAL pd context : sub-type : IPV4, ecd_marked:0, has_collapsed_ldi:0 collapse_bwalk_required:0, ecdv2_marked:0, HW Walk: LEAF: PI:0x3092d35d48 PD:0x3092d35df0 rev:1736174 type: IPV4 (0) TBL: 0xe0000002 LEAF location: EXT-TCAM FEC key: 0x17294000196e0 LWLDI: PI:0x3097d4cc78 PD:0x3097d4ccc0 rev:1736172 p-rev:1736165 ldi type:IMP FEC key: 0x17294000196e0 fec index: 0x2001df86(122758) num paths: 1 bkup paths: 1 IMP pattern:2 PI:0x3097d4cc78 PD:0x3097d4ccc0 rev:1736172 p-rev:1736165 dpa-rev:9996584 FEC key: 0x17294000196e0 fec index: 0x2001df86(122758) num paths: 1 bkup paths: 1 Path:0 fec index: 0x2001df86(122758) DSP fec index: 0x20000004(4), <<<<<<<<<<<<<<< IMP LWLDI hook to ECMP-FEC 4, not match to recursive LDI ECMP-FEC 5 below MPLS EEI push label: 27452 Path:1 fec index: 0x2001df87(122759) DSP fec index: 0x20000004(4), MPLS EEI push label: 27307 REC-SHLDI HAL PD context : ecd_marked:0, collapse_bwalk_required:0, load_shared_lb:0 RSHLDI: IMP FEC: PI:0x30988b5bb8 PD:0x30988b5c80 rev:2615481 dpa-rev:15048655 flag:0x5 FEC key: 0x14a84000196e0 fec index: 0x2001e40c(123916) num paths: 1 p-rev:2615472 Path:0 fec index: 0x2001e40c(123916) DSP fec index: 0x20000005(5), <<<<<<<<<< Recursive LDI IMP FEC hook to ECMP-FEC 5 47.15 . Path:1 fec index: 0x2001e40d(123917) DSP fec index: 0x20000005(5), BKUP-FRRNH: PI:0x309bc540d0 PD:0x309bc54199 rev: 2615472, p-rev: 2615379 2615471 2615453 , Flag:0x0 Indirection ECMP FEC key:0x2fbb82000196d0 fec index: 0x20000005(5) PROT-FRRNH: PI:0x309b54a810 PD:0x309b54a901 rev: 2615471 dpa-rev: 10141946 FRR Active: 0, Protect Hdl: 0x3097b33600, Id(H1,H2,H3): 0x4000001c,0x4000001c,0x4000001c, Status: 1 FRR EOD Pending: 0, Instance: 0 LEAF - HAL pd context : sub-type : MPLS, ecd_marked:0, has_collapsed_ldi:0 collapse_bwalk_required:0, ecdv2_marked:0, HW Walk: LEAF: PI:0x309d022098 PD:0x309d022140 rev:2615456 type: MPLS (2) TBL: 0 LEAF location: LEM FEC key: 00 LWLDI: PI:0x309814e028 PD:0x309814e070 rev:2615453 p-rev:1530126 ldi type:IMP_EOS0_EOS1 FEC key: 0x2d9804000196d0 fec index: 0x2001f9ad(129453) num paths: 1 bkup paths: 0 IMP pattern:3 PI:0x309814e028 PD:0x309814e070 rev:2615453 p-rev:1530126 dpa-rev:15048629 FEC key: 0x2d9804000196d0 fec index: 0x2001f9ad(129453) num paths: 1 bkup paths: 0 Path:0 fec index: 0x2001f9ad(129453) DSP: 0x12a MPLS encap key: 0xf1b000004001cf80 MPLS encap id: 0x4001cf80 Remote: 0 Label Stack(Top -> Bottom): { 27396 } dpa-rev: 15048628 SHLDI: PI:0x30988c0ce8 PD:0x30988c0db0 rev:1530126 dpa-rev:9310415 cbf_enabled:0 pbts_enabled:0 surpf_enable:0 flag:0x0 FEC key: 0x283c54000196d0 fec index: 0x2001fef6(130806) num paths: 1 bkup paths: 0 p-rev:1529685 Path:0 fec index: 0x2001fef6(130806) DSP:0x12a Dest fec index: 0x0(0) TX-NHINFO: PI: 0x3097a0d4c0 PD: 0x3097a0d550 rev:1529685 dpa-rev:9310076 Encap hdl: 0x3097a89480 Encap id: 0x40018003 Remote: 0 L3 int: 1051 flags: 0x3 npu_mask: 0x1 DMAC: e4:1f:7b:77:f6:4d Load distribution: 0 (refcount 5) Hash OK Interface Address 0 Y recursive 27406/0 // ECMP-FEC 4. is hooking to BGP PIC backup PE label 27388 ---------- // show control fia diagshell 0 "dump IHB_FEC_ECMP 4" loc IHB_FEC_ECMP.IHB0[4]: // show control fia diagshell 0 "diag alloc fec direct=1 from=0x1eadb to=0x1eadb" loc | 1ead8|mpls_lsr |1 | | | |SINGLE PORT| 298| 1cf76 | 0 | V | // show control fia diagshell 0 "diag pp lif_show id=1cf76 gl=1 type=out" loc entry ID 42482 contains the LIF: GLIF: 0x0001cf76 LLIF:0x0000a5f2 -> Bank:5 Offset:761 MPLS Encapsulation: nof_tunnels: 1 tunnel 1: tunnel_label: 27388 push_profile: 3 orientation NA out_vsi: 0 oam_lif_set: 0 outlif_profile: 0x11 Next_eep: 0x10004 // ECMP-FEC 5. is hooking to BGP PIC primary PE label 27396 ------------- show controllers fia diagshell 0 "dump IHB_FEC_ECMP 5" loc IHB_FEC_ECMP.IHB0[5]: // show control fia diagshell 0 "diag alloc fec direct=1 from=0x1f9ad to=0x1eadb" loc | 1f9ad|mpls_lsr |1 | | | |SINGLE PORT| 298| 1cf80 | 0 | V | // show control fia diagshell 0 "diag pp lif_show id=1cf80 gl=1 type=out" loc entry ID 48544 contains the LIF: GLIF: 0x0001cf80 LLIF:0x0000bda0 -> Bank:5 Offset:3792 MPLS Encapsulation: nof_tunnels: 1 tunnel 1: tunnel_label: 27396 push_profile: 3 orientation NA out_vsi: 0 oam_lif_set: 0 outlif_profile: 0x11 Next_eep: 0x10004