Symptom

Customer not able to run Speed Test. Device seems to be busy. When run from the cEdge or to the cEdge, the result is the same. It doesn't matter from which or to which color we run it. 30-Mar-2022 19:18:17,484 UTC INFO [] [SpeedTestManager] (pool-131-thread-1) || Creating netconf sourceIp 10.127.196.108, sourceColor biz-internet, destinationIp 10.127.196.1, destinationColor biz-internet port 80 sessionId 0248ffbd-fec3-4c84-9817-5f016abf1e0c targetDevice 10.127.196.1 30-Mar-2022 19:18:17,491 UTC INFO [US-vManage] [TroubleShootingManager] (default task-49640) |default| Request broadcasted for Source vManage 1.1.1.1, Destination vManage : 1.1.1.1, requestType: RequestType{requestType='TroubleShootingSpeedTest'} 30-Mar-2022 19:18:19,140 UTC INFO [] [DataCollectionManager] (device-event-data-sync-3) || already exist, not added ISR1100-6G-FGL2508L419 30-Mar-2022 19:18:19,169 UTC INFO [] [ClientSessionImpl] (sshd-SshClient[612a09fa]-nio2-thread-5) || requestNewKeysExchange(ClientSessionImpl[vmanage-admin@/10.127.192.84:830]) Initiating key re-exchange 30-Mar-2022 19:18:19,184 UTC INFO [] [SpeedTestManager] (pool-131-thread-1) || Creating netconf sourceIp 10.127.196.108, sourceColor biz-internet, destinationIp 10.127.196.1, destinationColor biz-internet port 80 sessionId 0248ffbd-fec3-4c84-9817-5f016abf1e0c targetDevice 10.127.196.108 30-Mar-2022 19:18:19,551 UTC INFO [] [DataCollectionManager] (device-event-data-sync-2) || already exist, not added ISR1100-6G-FGL2508L419 30-Mar-2022 19:18:19,627 UTC INFO [US-vManage] [TroubleShootingSessionManagerDAO] (default task-49642) |default| Error in session for session id: 0248ffbd-fec3-4c84-9817-5f016abf1e0c, Tenant default 30-Mar-2022 19:18:19,627 UTC ERROR [US-vManage] [RestfulDefaultExceptionMapper] (default task-49642) |default| Exception while processing request com.viptela.vmanage.server.troubleshooting.TroubleShootingException: Speed test in progress Full output in SpeedTestFailed.txt 30-Mar-2022 19:18:19,628 UTC ERROR [US-vManage] [RestfulDefaultExceptionMapper] (default task-49642) |default| Exception while processing request. Error code DEVICE_ERROR, Message : Device Error, Details: Speed test in progress 30-Mar-2022 19:18:19,773 UTC INFO [US-vManage] [TroubleShootingManager] (default task-49642) |default| Request broadcasted for Source vManage 1.1.1.1, Destination vManage : 1.1.1.1, requestType: RequestType{requestType='TroubleShootingSpeedTest'} 30-Mar-2022 19:18:19,797 UTC INFO [US-vManage] [TroubleShootingManager] (default task-49642) |default| Request broadcasted for Source vManage 1.1.1.1, Destination vManage : 1.1.1.1, requestType: RequestType{requestType='TroubleShootingSpeedTest'} 30-Mar-2022 19:18:19,856 UTC INFO [] [DataCollectionManager] (Thread-184) || ******* Requested device 10.127.196.205, Adding to dataCollectionQueue. 30-Mar-2022 19:18:25,797 UTC INFO [] [SpeedTestManager] (pool-131-thread-1) || Speed test enable successfully for session 0248ffbd-fec3-4c84-9817-5f016abf1e0c 30-Mar-2022 19:18:25,901 UTC INFO [] [SpeedTestManager] (pool-131-thread-1) || Creating netconf sourceIp 10.127.196.108, sourceColor biz-internet, destinationIp 10.127.196.1, destinationColor biz-internet port 80 sessionId 0248ffbd-fec3-4c84-9817-5f016abf1e0c targetDevice 10.127.196.1 30-Mar-2022 19:18:27,553 UTC INFO [] [ControlDeviceDataCollector] (device-data-collection-9725) || collectLinkData : DeviceNode DeviceNode{systemIp='10.127.200.148', deviceType='vedge', hostName='MXCHBRM-VEDGE01', personality='vedge', siteId=4110013, uuid='ISR1100-6G-FGL2508L3Z3', localSystemIP='10.127.200.148', baseVersion ='20.6.2'} , addLink false , expectedNumberOfConnections 4 , inMemory false , dataStoreType = DEFAULT 30-Mar-2022 19:18:27,553 UTC INFO [] [ControlDeviceDataCollector] (device-data-collection-9725) || number of connections 5 30-Mar-2022 19:18:27,555 UTC INFO [] [ControlDeviceDataCollector] (device-data-collection-9725) || vSmartConnCount 4, expectedNumberOfConnections 4 30-Mar-2022 19:18:27,602 UTC INFO [] [SpeedTestManager] (pool-131-thread-1) || Creating netconf sourceIp 10.127.196.108, sourceColor biz-internet, destinationIp 10.127.196.1, destinationColor biz-internet port 80 sessionId 0248ffbd-fec3-4c84-9817-5f016abf1e0c targetDevice 10.127.196.108 No stuck sessions in the vManage: neo4j@neo4j> match (n:vmanagedbTROUBLESHOOTINGSESSIONNODE) return n; +---+ | n | +---+ +---+ 0 rows available after 3 ms, consumed after another 0 ms

Conditions

One condition is that when speedtest running and the control connection disconnected or other network problem happened, the device not get the stop message from vManage, the speedtest is not really stopped at device side.

Workaround

No workaround found, although an NMS restart or cEdge reboot may resolve the issue the customer has.

Further Problem Description