Loading...
Loading...
All BFD sessions on a Port-Channel go down together and do not recover. %BFD-5-SESSION_STATE_DOWN: BFD session <id> to neighbor <addr> on interface Po1.100 has gone down. Reason: Control Detection Time Expired. Every session on that Port-Channel fails within a few hundred milliseconds of the first. BGP sessions protected by those BFD sessions are torn down, logged as due to bfd session down. BGP re-establishes on its own. BFD does not. The BFD sessions are recreated but never reach Up, and remain down until manual intervention. On the BGP neighbour the state is reported as Invalid: switch# show bgp vrf <vrf> ipv4 unicast neighbors <addr> BFD live-detection is configured and enabled, state is Invalid The configuration change that appears to cause the outage is typically unrelated to the affected interfaces, and may be in a different VRF.
Two separate events are required. Neither alone produces a symptom, and they may be separated by hours, days or weeks. Stage 1 - silent. Removal of any Port-Channel sub-interface clears the internal interface-manager value parent_pc_ifindex on every member link of that Port-Channel, including members unrelated to the sub-interface removed. switch(config)# no interface port-channel1.100 Nothing is observable. BFD stays Up, BGP stays Established, traffic is unaffected, and no syslog is generated beyond the sub-interface going down. Stage 2 - the outage. Deletion of any existing VLAN causes BFD to re-read the cleared value, and all BFD sessions on that Port-Channel fail. switch(config)# no vlan 100 The VLAN does not need to be VXLAN-mapped, have an SVI, or have any relationship to the Port-Channel. Deleting a VLAN that does not exist is a no-op and will not trigger it. Identifying an exposed switch. First obtain the source index (SI) of each Port-Channel member. The SI column is the value to use, and the Port-Channel's own If Index is the value each member should report: switch# slot 1 quoted 'show system internal iftmc info interface brief' Interface If Index LIF SI CR_FLAGS State Layer Mode #Mem ---------- ---------- ------- ------ --------- ----- ------ ----- ----- Po1 0x16000000 0x1004 0x2 INTF LIF UP L3 7 Eth1/1 0x1a000000 0x107f 0x1800 INTF LIF UP L3 Eth1/2 0x1a000200 0x1080 0x17fc INTF LIF UP L3 Then read each member using its SI value: switch# slot 1 quoted 'show system internal iftmc info src_index 0x1800 detail' | egrep 'parent_pc_ifindex' fbr_if : 0x0 parent_pc_ifindex : 0x16000000 <<< healthy fbr_if : 0x0 parent_pc_ifindex : 0x0 <<< exposed parent_pc_ifindex must equal the parent Port-Channel's own If Index - 0x16000000 in the example above. This value differs per switch and per Port-Channel, so read it from the interface table rather than assuming a fixed value. A member reading 0x0 is exposed. Every member must be checked. A switch showing 0x0 on any member is exposed while BFD, BGP and traffic all appear healthy. BFD does not need to be configured for Stage 1 to occur.
The cleared value is runtime state and is restored by reprogramming the member links. A reload also clears it, but is not required. Step 1. If BFD sessions on the Port-Channel are already Down, remove BFD from the BGP neighbours first, using the per-neighbour form: switch(config)# router bgp <asn> switch(config-router)# vrf <vrf> switch(config-router-vrf)# neighbor <addr> switch(config-router-vrf-neighbor)# no bfd Do not use no feature bfd; it removes the bfd keyword from every BGP neighbour statement and re-enabling the feature does not restore them. Step 2. Flap each Port-Channel member link individually, allowing each to rejoin before moving to the next: switch(config)# interface ethernet1/1 switch(config-if)# shutdown switch(config-if)# no shutdown Allow 15-20 seconds between members. One flap restores one member. All members must be flapped, including any that are administratively down but have an optic fitted. The Port-Channel remains up throughout. Step 3. Confirm every member reads 0x16000000 before proceeding. Step 4. Restore the BFD configuration if it was removed in Step 1. Sessions return without further intervention. Why Step 1 matters. If members are flapped while BFD sessions are live and returning, a session whose control packets hash onto the link being flapped will miss its detection window and tear down the BGP sessions it protects. Removing BFD first avoids this. Flapping a member link is not a zero-impact operation and should be scheduled accordingly.
Mechanism. This is an interface-manager (IFTMC) defect, not a BFD defect. Removal of a Port-Channel sub-interface causes IFTMC to clear parent_pc_ifindex for every member of the parent Port-Channel rather than only the interface being removed, and the value is not recalculated. BFD does not read this value during steady-state operation, so there is no symptom until something forces a re-read. VLAN deletion forces that re-read, after which BFD cannot program the session. Scope of impact. A session in this state continues to send BFD control packets but does not accept those received from the peer, so it never reaches Up. Data-plane forwarding is not affected by the cleared value itself - traffic loss occurs only during the BGP reconvergence that follows the BFD failure. Re-adding the removed sub-interface does not clear the condition. Restoring identical configuration leaves parent_pc_ifindex at 0x0. Only link reprogramming or a reload clears it. A non-disruptive ISSU (ND-ISSU) does not repair an exposed switch, including an ND-ISSU that installs the SMU for this defect. The switch arrives on fixed code still carrying the cleared value, and remains exposed until the member links are reprogrammed. An exposed switch has been observed to fail during the period following an ND-ISSU, with no configuration change performed. The upgrade completed successfully and BFD was Up afterwards; the sessions failed a number of minutes later. A short post-change verification would not have detected this. Recommended sequence: identify exposed switches, reprogram the member links, and upgrade only after confirming every member reads 0x16000000. Reproduction. Not every VLAN or VRF deletion triggers the issue. BFD remains Up after the VRF and interface deletions below, and fails only on the subsequent VLAN deletion: N9k(config)# no vrf A N9k(config)# no snmp-server context A vrf A N9k(config)# router bgp 64950 N9k(config-router)# no vrf A N9k(config-router)# no interface port-channel107.3749 N9k(config)# no interface Vlan201 N9k(config)# no interface Vlan15 N9k# show bfd neighbors vrf all OurAddr NeighAddr LD/RD RH/RS State Int Vrf 10.1.1.1 10.1.1.2 1090519044/1090519044 Up Up Po107.3747 B 10.2.2.1 10.2.2.2 1090519045/1090519045 Up Up Po107.3752 C N9k(config)# no vlan 15 <<< BFD now down on unrelated interfaces N9k# show bfd neighbors vrf all OurAddr NeighAddr LD/RD RH/RS State Int Vrf 10.1.1.1 10.1.1.2 1090519044/0 Down Down Po107.3747 B 10.2.2.1 10.2.2.2 1090519045/0 Down Down Po107.3752 C There is no syslog, alarm or counter indicating the exposed state. A direct read of parent_pc_ifindex is the only reliable check.
Cisco Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.