OPERATIONAL DEFECT DATABASE
...
...
VMware Aria Operations (Formerly known as vRealize Operations Manager) admin account shows an Incorrect User name/Password even after providing the correct credentials.After resetting the password using How to Reset the Admin Password in VMware Aria Operations , the issue reappears after a few successful logins. You can check the number of failed_attempts in /storage/vcops/user/conf/adminuser.properties, as per the example below: root@vrops810-11 [ ~ ]# cat /storage/vcops/user/conf/adminuser.properties #Properties for vCOps Admin user #Tue Feb 20 11:30:09 UTC 2024 hashed_password=<Password HASH removed> failed_attempts=5675 time_last_failure=1708428609792 username=admin
To locate the endpoint locking the admin account in VMware Aria Operations (Formerly known as vRealize Operations Manager)
This issue occurs when a user changes the VMware Aria Operations (Formerly known as vRealize Operations Manager) admin account password. The new password is not automatically updated in the endpoints that are linked to VMware Aria Operations, for example, Skyline Collector, Aria Operations for Logs (formerly vRealize Log Insight), etc.These endpoints make several API connect attempts to VMware Aria Operations (Formerly known as vRealize Operations Manager) using the old password. Since the password has been changed, the admin account keeps getting locked.
Due to this issue, the admin user is not able to access Aria Operations Product and Admin UIThis article does not apply to other products with Aria Operations in the name such as: Aria Operations for LogsAria Operations for Networks.
Steps to resolve the issue:1. SSH to the primary node of Aria Operations.2. The following commands can be used to help identify source of account lockout, command 3 is likely the most useful in most cases, even though it does not distinguish between admin or other accounts: Command 1:grep /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log*Example from lab server: 2024-02-20T11:52:17,086+0000 INFO [ajp-nio-127.0.0.1-8010-exec-3, 71EifC1H7nCAYsIGrOmdlRES9jEhPzfb] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=192.168.110.23, payload={"username":"admin","password":"*****"}] 2024-02-20T11:52:17,202+0000 INFO [ajp-nio-127.0.0.1-8010-exec-10, ivHLQF7tmorqYaAbt0J9T98EoWriqla7] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=192.168.110.23, payload={"username":"admin","password":"*****"}] 2024-02-20T11:52:17,359+0000 INFO [ajp-nio-127.0.0.1-8010-exec-2, 63ITvAa4VvTVephLHe4J0ABuFlx0VnoK] utils.HttpRequestLogger - After request [POST /suite-api/api/auth/token/acquire?_no_links=true, client=192.168.110.23, payload={"username":"admin","password":"*****"}] Notes: The above command will search all http_api.log* files for failed logins using admin account, but will give no indication as to whether these logins were successful or unsuccessful.To search the latest log file only, remove the * at the end of the log filename in the command Command 2:grep -a /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log | awk '{print $11}' | uniq -cExample from lab server: root@vrops810-11 [ ~ ]# grep -a /auth\/token\/acquire.*\"username\":\"admin\" /storage/log/vcops/log/http_api.log | awk '{print $11}' | uniq -c 1 client=192.168.110.205, 5 client=192.168.110.22, 2 client=192.168.110.23, 1 client=192.168.110.22, 1 client=192.168.110.23, 1 client=192.168.110.22, 6636 client=192.168.110.23, 16 client=192.168.110.21, 109 client=192.168.110.23, Notes: The above command will search the same log file as in command 1, and count how many logins there were per IP address. This is mostly useful to determine where admin logins are coming from and how many there were coming in from the same source.Repeating this command after a few seconds shows which endpoint is repeatedly attempting to log in.192.168.110.21, 192.168.110.22, and 192.168.110.23 in output above is from a 3 node Operations for Logs (LogInsight) cluster that was used to lock out admin account in our test environment. Command 3:grep -a /auth\/token\/acquire.* /storage/log/vcops/log/suite-api/localhost_access_log.txt | awk '{print $3}' | sort | uniq -cExample from lab server: root@vrops810-11 [ ~ ]# grep -a /auth\/token\/acquire.* /storage/log/vcops/log/suite-api/localhost_access_log.txt | awk '{print $3}' | sort | uniq -c 10839 127.0.0.1 1 192.168.110.205 16 192.168.110.21 7 192.168.110.22 6784 192.168.110.23 Notes: The above command will search /storage/log/vcops/log/suite-api/localhost_access_log.txt log file.Log file reports the HTTP codes (200/401/etc) as well as IP address of endpoint.As with command 2 above, repeat after a few seconds to determine IP where the count increases.This command does not distinguish between admin account or other accounts Example output from /storage/log/vcops/log/suite-api/localhost_access_log.txt: Successful login: [20/Feb/2024:12:17:28 +0000] 192.168.110.23 - - "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 200 204 19 Failed login: [20/Feb/2024:11:29:31 +0000] 192.168.110.23 - - "POST /suite-api/api/auth/token/acquire?_no_links=true HTTP/1.1" 401 125 27 Note the 200 HTTP code and the 401 HTTP code in the example log output above. Summary: The purpose of the above commands is to help identify which remote system is attempting to log in, and is the likely cause of the account lockout.Note that even though an endpoint is unable to log in, resulting in an increase in failed login attempts, this may not be the cause of the initial lockout. However, in most cases we see that it's the integration with endpoints that are failing to log in that is the cause of the account lockout, after Admin password was changed on Aria Operations.If Aria Operations is behind a load balancer, and endpoint use VIP to access Aria Operations, the IP shown will normally refer to the LB itself. You will need to investigate the LB to identify the source.You can ignore logins from 127.0.0.1, other analytics nodes, and Cloud Proxies/Remote Collectors that have joined the cluster. We are normally only interested in addresses that are not members of the cluster. Command 2 and command 3 above are the most useful to determine where login attempts are coming from, and repeating the commands usually show an increased count from when you repeat the command. 3. After you've identified the source of the lockouts, and changed the integration password on the endpoint, follow KB Log in to Admin and Product UI as admin fails in vRealize/Aria Operations to reset the lockout on the admin account.
For information on resetting the Admin password, see How to reset the admin password in vRealize Operations (2078313).For information on resetting the Admin account lockout, see Log in to Admin and Product UI as admin fails in vRealize/Aria Operations (2131633) .
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
