Symptom

After sending ARP/ND request for 1600 point-to-point neighbors reachable over 1600 subint, the following logs begin appearing: LC/0/0/CPU0:Dec 6 19:08:47.750 UTC: fib_mgr[243]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=FIB_DATA_TYPE_NHINFO[ptr=0x309e263670,refc=0x5,flags=0x8000013] Action=MODIFY Proto=mpls. Cerr='OFA' detected the 'informational' condition 'OFA Infra DB': No such file or directory : fib_mgr : (PID=3693) : -Traceback= 7f2201a81178 55658ee8707d 55658ecf4c66 55658ee6ea29 55658ee7124e 55658ed06b84 55658ed06d65 55658ee6d71d 55658ed76a89 55658ed720a6 55658ee7ad98 55658ed66e2f 55658ed69ac4 55658ee7b3ff 55658eeb1531 55658ee7ad98 LC/0/0/CPU0:Dec 6 19:09:02.699 UTC: fib_mgr[243]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=FIB_DATA_TYPE_NHINFO[ptr=0x309dcdeaf0,refc=0x5,flags=0x8000013] Action=MODIFY Proto=mpls. Cerr='OFA' detected the 'warning' condition 'OFA DB delete pending': Text file busy : fib_mgr : (PID=3693) : -Traceback= 7f2201a81178 55658ee8707d 55658ecf4c66 55658ee6ea29 55658ee7124e 55658ed06b84 55658ed06d65 55658ee6d71d 55658ed76a89 55658ed720a6 55658ee7ad98 55658ed66e2f 55658ed69ac4 55658ee7b3ff 55658eeb1531 55658ee7ad98 ===== TRIGGER #1 ===== Logs appear as soon as GRE tunnels are added to MPLS static process (even without any static LSP entry: RP/0/RP0/CPU0:A08-01-NCS5508-CX1#show run mpls static Tue Nov 23 21:38:13.852 UTC mpls static interface tunnel-ip1001 ! ===== TRIGGER #2 ===== After Bundle-Ether member flap, logs are flooding console: RP/0/RP0/CPU0:Dec 3 06:22:30.959 UTC: BM-DISTRIB[1216]: %L2-BM-6-ACTIVE : HundredGigE0/0/1/5 is no longer Active as part of Bundle-Ether30 (Wait-while timer is running) RP/0/RP0/CPU0:Dec 3 06:22:32.966 UTC: BM-DISTRIB[1216]: %L2-BM-6-ACTIVE : HundredGigE0/0/1/5 is Active as part of Bundle-Ether30 ! LC/0/0/CPU0:Dec 3 06:24:07.721 UTC: fib_mgr[243]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=FIB_DATA_TYPE_NHINFO[ptr=0x309e262ed8,refc=0x3,flags=0x8000011] Action=MODIFY Proto=mpls. Cerr='OFA' detected the 'informational' condition 'OFA Infra DB': No such file or directory : fib_mgr : (PID=3693) : -Traceback= 7f2201a81178 55658ee8707d 55658ecf4c66 55658ecf5ba7 55658ee706a2 55658eeb5d56 7f21fdaeb6bb 7f21fdaeb8c4 7f21ffd27c10 7f21ffd354f4 55658ec406d7 ! LC/0/0/CPU0:Dec 3 06:24:19.221 UTC: fib_mgr[243]: %ROUTING-FIB-3-PLATF_UPD_FAIL : FIB platform update failed: Obj=FIB_DATA_TYPE_NHINFO[ptr=0x3090775ef0,refc=0x3,flags=0x8000011] Action=MODIFY Proto=mpls. Cerr='OFA' detected the 'warning' condition 'OFA DB delete pending': Text file busy : fib_mgr : (PID=3693) : -Traceback= 7f2201a81178 55658ee8707d 55658ecf4c66 55658ecf5ba7 55658ee706a2 55658eeb5d56 7f21fdaeb6bb 7f21fdaeb8c4 7f21ffd27c10 7f21ffd354f4 55658ec406d7 Seeing all MPLS-related destinations stuck at retry-db: RP/0/RP0/CPU0:A08-05-5501-sPF2#show cef retry-db Mon Dec 6 23:09:08.671 UTC RP/0/RP0/CPU0:A08-05-5501-sPF2#show cef retry-db location 0/0/CPU0 Mon Dec 6 23:09:11.270 UTC -------------------- ---------- ----- ---------- --------------------- Obj-Type Retry Retry Scheduling Timestamp Flags Count Class -------------------- ---------- ----- ---------- --------------------- NHINFO 0x4 20062 Slow Dec 6, 2021 23:09:05 Error code: 0x7f21, 'Subsystem(0)' detected the 'success' condition 'Code(63)' NHINFO nh:0x309e263670 ifh:0x2000c754 nh:X.X.X.X/32 nhtype:tx linkt:link_MPLS nh_proto:IPv4 Not stuck for all 1638 entries, but only for 619: RP/0/RP0/CPU0:A08-05-5501-sPF2#show cef retry-db location 0/0/CPU0 | in "NHINFO nh" | util wc -l Mon Dec 6 23:11:18.580 UTC 619 RP/0/RP0/CPU0:A08-05-5501-sPF2#show im database ifhandle 0x2000c754 Mon Dec 6 23:09:43.993 UTC View: OWN - Owner, L3P - Local 3rd Party, G3P - Global 3rd Party, LDP - Local Data Plane GDP - Global Data Plane, RED - Redundancy, UL - UL Node 0/RP0/CPU0 (0x1000) Interface Bundle-Ether30.1270, ifh 0x2000c754 (up, 1522) Interface flags: 0x00000000000205d7 (REPLICATED|IFINDEX |SUP_NAMED_SUB|BROADCAST|VIRTUAL|CONFIG|VIS|DATA |CONTROL) Encapsulation: dot1q Interface type: IFT_VLAN_SUBIF Control parent: Bundle-Ether30 Data parent: Bundle-Ether30 Views: UL|GDP|G3P|L3P|OWN Protocol Caps (state, mtu) -------- ----------------- None netflow_in (up, 1522) None vlan_jump (up, 1522) None dot1q (up, 1522) arp arp (up, 1504) ipv4 ipv4_acl_in (up, 1504) ipv4 ipv4 (up, 1504) mpls mpls (up, 1504) ipv6 ipv6_pfilter_in (up, 1504) ipv6 ipv6_preswitch (up, 1504) ipv6 ipv6 (up, 1504) CEF entries have not been updated on RP nor LC since issue appeared: RP/0/RP0/CPU0:A08-05-5501-sPF2#show cef X.X.X.X/32 detail Tue Dec 7 00:24:08.898 UTC <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< X.X.X.X/32, version 0, not-preferred-src, internal 0x1020001 0xf0 (ptr 0x9faab410) [1], 0x0 (0x89df4f50), 0x0 (0x0) Updated Dec 3 07:13:49.683 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< local adjacency to Bundle-Ether30.1270 Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 15 gateway array (0x9e1ad900) reference count 1, flags 0x0, source aib (3), 0 backups [2 type 3 flags 0x8401 (0x9e4c7138) ext 0x0 (0x0)] LW-LDI[type=3, refc=1, ptr=0x89df4f50, sh-ldi=0x9e4c7138] gateway array update type-time 1 Dec 3 07:13:49.683 LDI Update time Dec 3 07:13:49.683 LW-LDI-TS Dec 3 07:13:49.683 via X.X.X.X/32, Bundle-Ether30.1270, 3 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 NHID 0x0 [0x9fbe47d8 0x0] next hop X.X.X.X/32 local adjacency Load distribution: 0 (refcount 2) Hash OK Interface Address 0 Y Bundle-Ether30.1270 X.X.X.X RP/0/RP0/CPU0:A08-05-5501-sPF2#show cef X.X.X.X/32 detail location 0/0/CPU0 Tue Dec 7 00:24:12.836 UTC <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< X.X.X.X/32, version 0, not-preferred-src, internal 0x1020001 0xf0 (ptr 0x9cb70b78) [1], 0x0 (0x9ee451a8), 0x0 (0x0) Updated Dec 3 07:13:49.587 <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< local adjacency to Bundle-Ether30.1270 Prefix Len 32, traffic index 0, Adjacency-prefix, precedence n/a, priority 15 gateway array (0x9b43cf80) reference count 1, flags 0x0, source aib (3), 0 backups [2 type 3 flags 0x8401 (0x9b70f508) ext 0x0 (0x0)] LW-LDI[type=3, refc=1, ptr=0x9ee451a8, sh-ldi=0x9b70f508] gateway array update type-time 1 Dec 3 07:13:49.587 LDI Update time Dec 3 07:13:49.587 LW-LDI-TS Dec 3 07:13:49.587 via X.X.X.X/32, Bundle-Ether30.1270, 3 dependencies, weight 0, class 0 [flags 0x0] path-idx 0 NHID 0x0 [0xa2189880 0x0] next hop X.X.X.X/32 local adjacency Load distribution: 0 (refcount 2) Hash OK Interface Address 0 Y Bundle-Ether30.1270 X.X.X.X

Conditions

Issue appears after sending ARP/ND request for 1600 point-to-point neighbors reachable over 1600 subint. Issue has been seen with 1000 and 1600 subint scale

Workaround

None

Further Problem Description