Symptom

BGP went into Mutex state upon removing the Un-Used VRF's which are called under BGP, Static and removing vrf globally in one single commit. This also led to Out Queue stuck on 1K RR. RP/0/RSP0/CPU0:RTR#sh processes blocked Wed Jan 26 14:11:32.591 IST Jid Pid Tid ProcessName State TimeInState Blocked-on 80245 14709 14709 bgp_show Reply 0000:01:19.0150 5035 sysdb_mc 1016 3082 3119 isis Reply 3769:30:21.0939 9849 lspv_server 1014 10368 10490 isis Reply 5435:38:28.0266 9849 lspv_server 1013 10367 10492 isis Reply 5435:38:28.0248 9849 lspv_server 1015 10369 10469 isis Reply 5435:38:28.0384 9849 lspv_server 1087 10346 10602 bgp Mutex 0025:02:59.0717 0x7f95bc3de178 10600 bgp rn_sync_tree_lock:147 1087 10346 10601 bgp Mutex 0024:53:49.0559 0x7f95bc3de178 10600 bgp rn_sync_tree_lock:147 1087 10346 10598 bgp Mutex 0025:03:52.0211 0x7f95bc3de178 10600 bgp rn_sync_tree_lock:147 1087 10346 10597 bgp Mutex 0025:03:52.0171 0x7f95bc3de178 10600 bgp rn_sync_tree_lock:147 1087 10346 10557 bgp Reply 5435:38:27.0426 9849 lspv_server 1087 10346 10556 bgp Mutex 0025:03:52.0212 0x7f95bc3de178 10600 bgp rn_sync_tree_lock:147 1087 10346 10404 bgp Mutex 0025:00:52.0618 0x5603645b7e60 10599 bgp bgp_updgrp_table_close:7837 RP/0/RSP0/CPU0:RTR#sh configuration commit changes 1000000332 !! IOS XR Configuration 6.6.3 no vrf no vrf no vrf no vrf ... router static no vrf no vrf no vrf no vrf ... ! router bgp no vrf no vrf no vrf no vrf .. RP/0/RSP1/CPU0:2022 Jan 25 13:15:39.876 IST: bgp[1087]: %ROUTING-BGP-3-NPL_TIMEOUT : NPL timeout - BGP will reinitialize NSR RP/0/RSP0/CPU0:2022 Jan 25 13:18:39.987 IST: bpm[1102]: %ROUTING-BGP-5-ASYNC_IPC_STATUS : bpm-default:(S)inst-id 0, Connection Closed RP/0/RSP0/CPU0:2022 Jan 25 13:18:42.663 IST: bpm[1102]: %ROUTING-BGP-5-ASYNC_IPC_STATUS : bpm-default:(S)inst-id 0, Connection Open RP/0/RSP1/CPU0:2022 Jan 25 13:18:43.469 IST: bgp[1087]: %ROUTING-BGP-5-ASYNC_IPC_STATUS : default, process instance 1:(S)inst-id 0, Connection Establised RP/0/RSP1/CPU0:2022 Jan 25 13:18:43.622 IST: bgp[1087]: %ROUTING-BGP-5-ASYNC_IPC_STATUS : default:(S)inst-id 0, Initial Config Done RP/0/RSP0/CPU0:2022 Jan 25 14:34:08.681 IST: sysdb_shared_nc[298]: %SYSDB-SYSDB-6-TIMEOUT_EDM : EDM request for 'oper/ip-bgp/gl/instance/default/act/shared/vrf/default/afi/' from 'bgp_show' (jid 68783, node 0/RSP0/CPU0). No response from 'bgp' (jid 1087, node 0/RSP0/CPU0) within the timeout period (100 seconds) RR#sh ip bgp update-group BGP version 4 update-group 1, internal, Address Family: IPv4 Unicast BGP Update version : 1/13420768, messages 1000, active RGs: 1 Route-Reflector Client Advertise additional-path: all Community attribute sent to this neighbor Extended-community attribute sent to this neighbor Sending Prefix & Label Topology: global, highest version: 13420768, tail marker: 13420768 Format state: Current blocked (no message space, last no message space) Refresh blocked (no message space, last not in list) Update messages formatted 16087461, replicated 2511088233, current 1000, refresh 0, limit 1000, mss 980, SSO is disabled <<<< Queue 1000 for current & limit is showing stuck Number of NLRIs in the update sent: max 337, min 0 Minimum time between advertisement runs is 0 seconds Has 164 members: YY.YY.YY.YY XX.XX.XX.XX <<<<<< Both Impacted PE due to Mutex and PE which are not receiving the updates are in same update-group

Conditions

Removing VRF Globally and from router bgp, static in one commit

Workaround

BGP process restart

Further Problem Description