Symptom

ASA may experience a High CPU caused by stuck SSH sessions running despite the fact that originating connections are no longer place. Depending on the number of SSH sessions attempted, maximum concurrent number of SSH sessions is 5, you may be unable to SSH into the ASA in case you have 5 stuck SSH sessions. Symptoms seen on affected devices: 1 - ASA showing High CPU: > On single core platforms: asa# show cpu usage CPU utilization for 5 seconds = 97%; 1 minute: 97%; 5 minutes: 97% > On multi-core platforms, High CPU will be seen on few particular cores: asa# show cpu detailed Break down of per-core data path versus control point cpu usage: Core 5 sec 1 min 5 min Core 0 51.0 (1.0 + 50.0) 51.1 (1.0 + 50.1) 51.1 (1.0 + 50.0) <<<<<< Core 1 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 2 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 3 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 4 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 5 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 6 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 7 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 8 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 9 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 10 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) 1.8 (1.8 + 0.0) Core 11 51.4 (1.4 + 50.0) 51.0 (1.3 + 49.8) 51.0 (1.2 + 49.8) <<<<<< 2 - Show processes output displaying concurrent SSH sessions, even if there are not the same number of active ones: asa# show processes cpu-usage sorted non-zero ASLR enabled, text region 559d7d5f2000-559d81f75345 PC Thread 5Sec 1Min 5Min Process - - 26.8% 27.7% 28.1% DATAPATH-0-1673 0x0000559d7f891dad 0x00007f858d649b20 16.1% 15.8% 15.7% ssh 0x0000559d7f891dad 0x00007f858d64c6a0 13.5% 13.3% 13.3% ssh 0x0000559d7f891dad 0x00007f858d648ca0 11.1% 10.9% 10.9% ssh 0x0000559d7f891dad 0x00007f858d64a600 11.0% 10.8% 10.8% ssh 0x0000559d7f891dad 0x00007f858d6476e0 10.7% 10.6% 10.6% ssh 0x0000559d7fef99a4 0x00007f858d6e0d00 4.0% 4.1% 4.1% Logger 0x0000559d7e7f5a95 0x00007f858d6d00c0 2.3% 2.4% 2.4% CP Processing 0x0000559d7e322516 0x00007f858d6ee680 0.9% 0.9% 0.9% Config History Thread 0x0000559d7ff9f633 0x00007f858d6c2e80 0.1% 0.0% 0.0% snmp 3 - Show resource usage showing device (or contexts) with high number of concurrent sessions for line "SSH Server": asa# show resource usage resource ssh Resource Current Peak Limit Denied Context SSH Server 1 2 5 0 admin SSH Server 0 1 5 0 one SSH Server 0 6 5 0 two SSH Server 0 1 5 0 three SSH Server 5 6 5 0 four <<<<<<< 4 - Despite command output above, no actual SSH connection is found active on context in question: asa/four/act# show conn all protocol tcp port 22 detail 1203 in use, 1902 most used asa/four/act# show ssh sessions [no sessions present]

Conditions

Problem first seen on: > ASA devices upgraded from any code to release 9.12.X > ASA configured with an empty enable password, which triggers the "enable password" enforcement when attempting SSH to the device > On next SSH session established after the upgrade, ASA will present the following prompt: ASA > ena The enable password is not set. Please set it now. Enter Password: Repeat Password: > If user fails to enter a matching password on the prompt above, and terminates the SSH session abruptly, the SSH session in question is not grecefully terminated by the ASA and keeps running on the background > This situation was also seen by customers that have monitoring tools connecting to the ASA and running commands scripts. As those scripts are not expecting to hit such password prompt, their execution fails and the SSH sessions will get stuck as well. >>> Related feature introduced on 9.12: https://www.cisco.com/c/en/us/td/docs/security/asa/asa912/release/notes/asarn912.html#reference_u13_j2w_gfb "enable password change now required on a login The default enable password is blank. When you try to access privileged EXEC mode on the ASA, you are now required to change the password to a value of 3 characters or longer. You cannot keep it blank. The no enable password command is no longer supported. At the CLI, you can access privileged EXEC mode using the enable command, the login command (with a user at privilege level 2+), or an SSH or Telnet session when you enable aaa authorization exec auto-enable . All of these methods require you to set the enable password. This password change requirement is not enforced for ASDM logins. In ASDM, by default you can log in without a username and with the enable password."

Workaround

1 - Reload the ASA to release the SSH sessions from the box. 2 - Post reload immediately SSH to the ASA and set a new enable password on box by: 2.1 - Change to enable mode: ASA > enable 2.2 - Enter a new password twice on the following prompts displayed: The enable password is not set. Please set it now. Enter Password: Repeat Password: 3 - Save the ASA configuration by running "write mem" Note: - For multiple context ASAs, repeat the process for every single context enabled for SSH and with empty enable password.

Further Problem Description