Symptom
While using Callhome and Smart Software Licensing feature, callhome MTS messages can get stuck due to lack of DNS resolution.
Issue can be seen while trying to display running-configuration or any other commands that require information from callhome process like "show callhome".
Conditions
Callhome and Smart Software Licensing configured on the switch.
Workaround
Reload switch to clear MTS issue and quickly change DNS configuration on the switch to make DNS queries resolvable.
Further Problem Description
We can see that switch is trying to resolve DNS name related to callhome service:
Nexus# ethanalyzer local interface inband display-filter dns limit-captured-frames 100000
Capturing on inband
2020-07-24 08:01:37.454260 10.10.10.10 -> 10.10.0.1 DNS Standard query AAAA tools.cisco.com
2020-07-24 08:01:42.462821 10.10.10.10 -> 10.10.0.1 DNS Standard query AAAA tools.cisco.com
2020-07-24 08:01:47.469911 110.10.10.10 -> 10.10.0.1 DNS Standard query AAAA tools.cisco.com.fra1
2020-07-24 08:01:52.494684 10.10.10.10 -> 10.10.0.1 DNS Standard query AAAA tools.cisco.com.fra1
There is no reply from DNS server for some reason (misconfiguration, DNS server not responsive...)
This will cause MTS messages being stuck:
Nexus# sh system internal mts buffer sum
* recv_q: not received yet (slow receiver)
* pers_q/npers_q/log_q: received not dropped (leak)
node sapno recv_q pers_q npers_q log_q app/sap_description
sup 8512 1 0 0 0 licmgr/callhome lib SL session
sup 7937 1 0 0 0 licmgr/callhome lib SL session
sup 66 0 1 0 0 callhome/callhome server <<<<<<<<<<<<<<<<<<<<<
sup 284 0 6 0 0 netstack/TCPUDP process client MTS queue
There will be delay while displaying running configuration as there is no reply from SAP 66 (callhome server).
Example:
Nexus# show running-config
The following SAPs did not respond within the expected timeframe
Pending SAPS:66
Printing Ascii configuration for remaining SAPs
!Command: show running-config
