General
APIC controller attached to 1st generation top of rack switches (such as N9K-C9396PX, N9K-C93128TX or C9372PX ) got diverged after APIC controller firmware upgrade
Symptom
Cisco APICs cannot ping each other through inband management.
LLDP neighbors on the Cisco APIC sees the directly-connected leaf switch as a neighbor.
The 'show lldp neighbor' command on the leaf switch sees the directly-attached Cisco APIC as a neighbor.
Pinging the infra bridge domain gateway 10.0.0.30 from the Cisco APIC does not work.
Conditions
This issue is seen rarely during upgrade of Cisco APICs and when the Cisco APICs are connected to leaf switches that do not have a suffix (such as -EX or -FX) in the product ID (such as N9K-C9396PX, N9K-C93128TX, or N9K-C9372PX).
A possible condition is that the firmware for the Cisco APICs and switches are upgraded in parallel.
Workaround
Upgrade to a software release that includes the software fix.
Upgrade the firmware of all Cisco APICs first and wait for the upgrade to complete before starting the firmware upgrade of the switches to minimize the occurrence of this issue.
Further Problem Description
Recovery:
Power cycle or Issue reload on top of rack switchs connected to APIC controller from where ping to 10.0.0.30 is not working.
And also reboot APIC controller where ping to infra BD gateway 10.0.0.30 is not working.
