Symptom
signal abort type of crash
boot loop
these messages
Oct 5 20:43:42.947: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'iosd <-- ftm ipsec' has taken 4947 msec (runtime: 0 msec) to process a 'unknown' message
Oct 5 20:43:47.947: %IOSXE_INFRA-6-PROCPATH_CLIENT_HOG: IOS shim client 'iosd <-- ftm ipsec' has taken 9947 msec (runtime: 0 msec) to process a 'unknown' message
Oct 5 20:43:37.996: SDWAN ERR: iosd_ftm_sdwan_msg_handler: ipsec msg handling failed
Oct 5 20:43:37.997: SDWAN ERR: iosd_ftm_sdwan_msg_handler: ipsec msg handling failed
Oct 5 20:43:37.998: SDWAN ERR: iosd_ftm_sdwan_msg_handler: ipsec msg handling failed
Oct 5 20:43:37.999: SDWAN ERR: iosd_ftm_sdwan_msg_handler: ipsec msg handling failed
Conditions
following scenarios were seen iosxe upgrades running sdwan code in 8500-12X platform
1.-from 17.6.1a to 17.6.3a
2.-from 17.6.3a to 17.9.4
-full mesh sdwan topology
-one of the routers that crashed was hub
Workaround
shutdown neighboring connected interfaces attached to LAN (services) ports, facing the LAN side
this was found as coincidence in an attempt to avoid neighbor devices flapping
Further Problem Description
the upgrade trigger at this point is not clear since sometimes the issue does not happen.
-after they shut down the LAn ports, everything remained stable, later on, the "unshut" those ports that were shut down as per workaround enclosure and those ports remained up, so everything fine and recovered afterwards.
