Symptom
[+] Current CPU load is mainly relating to stats-client and tahusd, However its not just limited to Stats-client or tahusd , different processes are causing high CPU.
Switch# show system internal proc cpu | ex 0.0
top - 13:53:30 up 118 days, 11:25, 4 users, load average: 12.35, 11.28, 10.50
Tasks: 532 total, 7 running, 420 sleeping, 0 stopped, 105 zombie
MiB Mem : 32044.0 total, 17986.5 free, 4552.8 used, 9504.8 buff/cache
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
26432 root -2 0 2979508 1.2g 329360 R 172.7 3.8 21022:49 tahusd
27537 root 25 5 467568 31456 20720 R 40.9 0.1 12528:49 stats_client
26148 root 20 0 658612 158420 52416 S 22.7 0.5 3161:09 mfdm
25736 root 20 0 4067244 115320 50160 S 20.5 0.4 874:34.38 sla_s
[+] A few MTS messages were stuck for a very long time.
`show system internal mts buffers summary`
* recv_q: not received yet (slow receiver)
* pers_q/npers_q/log_q: received not dropped (leak)
node sapno recv_q pers_q npers_q log_q app/sap_description
sup 38542 2 0 0 0 service_notif/udp socket
sup 38541 1 0 0 0 service_notif/udp socket
sup 284 0 6 0 0 netstack/TCPUDP process client MTS queue
`show system internal mts buffers detail`
**Fast Sap Buffers are not displayed below**
Node/Sap/queue Age(ms) SrcNode SrcSAP DstNode DstSAP OPC MsgId MsgSize RRToken Offset
sup/38542/recv 4421 0x101 284 0x101 38542 0 0x4e93f5f7 84 0x4e93f5f7 0xfa21404
sup/38542/recv 1240 0x101 284 0x101 38542 0 0x4e93fc7a 241 0x4e93fc7a 0xfa21004
sup/38541/recv 4428 0x101 284 0x101 38541 0 0x4e93f5f1 104 0x4e93f5f1 0xfa21604
sup/284/pers 100221753 0x101 5355 0x101 284 86017 0x6da1ef87 6228 0x6da1ef87 0xfa32004 <<<<<<<<<<<<-- Stuck for More than 26+ hrs
sup/284/pers 529660 0x101 4544 0x101 284 86017 0x6eab068c 6228 0x6eab068c 0xfa22004
sup/284/pers 36011 0x101 22871 0x101 284 86017 0x6eaca6a6 6228 0x6eaca6a6 0xfa34004
sup/284/pers 13381 0x101 8644 0x101 284 86017 0x6ead5a2e 6228 0x6ead5a2e 0xfa24004
sup/284/pers 1645 0x101 5654 0x101 284 86017 0x6ead7d8d 6228 0x6ead7d8d 0xfa30004
sup/284/pers 399 0x101 5722 0x101 284 86017 0x6ead7e08 6228 0x6ead7e08 0xfa26004
Conditions
N9K Switch running on code 10.2(4)
