Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Multiple VMware Cloud Foundation instances in the same SSO Domain.SSO Administrator password "administrator@vsphere.local" rotation (manually or scheduled) on one of the VMware Cloud Foundation instances.SDDC manager in second instance becomes inaccessible with "administrator@vsphere.local" or any other privileged user.SDDC manager database in the second VMware Cloud Foundation Instance has incorrect password for "administrator@vsphere.local" .
This article provides information about the impact of changing the SSO administrator password "administrator@vsphere.local" .The article also provides steps to remediate the issue through steps to update the "administrator@vsphere.local" on the second instance.
Multiple VMware Cloud Foundation instances can be joined together in a single SSO domain and each VMware Cloud Foundation instance has an entry for "administrator@vsphere.local" in the associated SDDC Manager database.Since all VCF instances are joined together in same PSC SSO, the credentials for SSO account "administrator@vsphere.local" needs to be manually updated in each additional VMware Cloud Foundation instance after a password rotation in the primary instance.The credential for "administrator@vsphere.local" can be rotated and/or updated from the primary VMware Cloud Foundation instance, which results in additional instances having a stale password which renders the SDDC Manager UI inaccessible.This requires a Remediate operation in the additional VMware Cloud Foundation instances immediately so that password stored in secondary instances is updated to reflect the change.
To update the SSO Administrator Password on an additional instance, follow the below steps: On the primary VMware Cloud Foundation instance, connect to SDDC Manager via ssh.su to rootRun the lookup_passwords command using admin@local account or any other SSO account which has the ADMIN role in SDDC Manager, and note down the password for "administrator@vsphere.local" vcf@sddc-manager [ ~ ]$ lookup_passwords -e PSCPassword lookup operation requires ADMIN user credentials. Please refer VMware Cloud Foundation Operations and Administration Guide for setting up ADMIN user.Enter page number (optional):Enter page size (optional, default=50):Enter Username: vcf-secure-user@vsphere.localEnter Password: PSC identifiers: x.x.x.x,vcenter-1.vrack.vsphere.local workload: sddcId-1001 username: administrator@vsphere.local password: ********* --->New Password type: SSO account type: SYSTEMNote: X.X.X.X is the vCenter Server IP address On the additional VMware Cloud Foundation instance, connect to SDDC Manager via ssh.su to rootUsing admin@local account or any other SSO account which has the ADMIN role in SDDC Manager, create an API access_token vcf@sddc-manager [ ~ ]$ curl -d '{"username" : "vcf-secure-user@vsphere.local", "password" : "********"}' -H "Content-Type: application/json" -X POST localhost/v1/tokens -k | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 1798 0 1727 100 71 5588 229 --:--:-- --:--:-- --:--:-- 5818{ "accessToken": "eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI5ZGYxNzU0Yi04ZDc3LTQzODMtYTEyZC0wOTdkMzJiZTEzMGQiLCJpYXQiOjE2MjczODIyODksInN1YiI6InZjZi1zZWN1cmUtdXNlckB2c3BoZXJlLmxvY2FsIiwiaXNzIjoidmNmLWF1dGgiLCJhdWQiOiJzZGRjLXNlcnZpY2VzIiwibmJmIjoxNjI3MzgyMjg5LCJleHAiOjE2MjczODU4ODksInVzZXIiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsIm5hbWUiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsInNjb3BlIjpbIlNERENfRkVERVJBVElPTl9XUklURSIsIkFWTl9XUklURSIsIlNERENfTUFOQUdFUl9SRUFEIiwiQ0VSVF9XUklURSIsIkNPTVBPU0FCSUxJVFlfV1JJVEUiLCJMSUNFTlNFX0tFWV9SRUFEIiwiQ09NUE9TQUJJTElUWV9SRUFEIiwiRURHRV9DTFVTVEVSX1dSSVRFIiwiVVNFUl9SRUFEIiwiQ1JFREVOVElBTF9XUklURSIsIkJBQ0tVUF9DT05GSUdfUkVBRCIsIkNMVVNURVJfV1JJVEUiLCJBVk5fUkVBRCIsIlZBU0FfUFJPVklERVJfUkVBRCIsIkRPTUFJTl9XUklURSIsIkNFSVBfUkVBRCIsIlNPU19XUklURSIsIlNERENfTUFOQUdFUl9XUklURSIsIk5UUF9XUklURSIsIkRFUE9UX0NPTkZJR19XUklURSIsIkRFUE9UX0NPTkZJR19SRUFEIiwiSE9TVF9XUklURSIsIkJBQ0tVUF9SRVNUT1JFX1JFQUQiLCJDRVJUX1JFQUQiLCJVU0VSX1dSSVRFIiwiVVBHUkFERV9SRUFEIiwiT1RIRVJfUkVBRCIsIlNPU19SRUFEIiwiU0VDVVJJVFlfQ09ORklHX1JFQUQiLCJDUkVERU5USUFMX1JFQUQiLCJIT1NUX1JFQUQiLCJDRUlQX1dSSVRFIiwiT1RIRVJfV1JJVEUiLCJMSUNFTlNFX0tFWV9XUklURSIsIkNBX1JFQUQiLCJORVRXT1JLX1BPT0xfV1JJVEUiLCJXQ1BfUkVBRCIsIkJBQ0tVUF9SRVNUT1JFX1dSSVRFIiwiTlRQX1JFQUQiLCJFREdFX0NMVVNURVJfUkVBRCIsIkJBQ0tVUF9DT05GSUdfV1JJVEUiLCJXQ1BfV1JJVEUiLCJTRVJWSUNFX0FDQ09VTlRfV1JJVEUiLCJORVRXT1JLX1BPT0xfUkVBRCIsIkNBX1dSSVRFIiwiQ0xVU1RFUl9SRUFEIiwiVkFTQV9QUk9WSURFUl9XUklURSIsIkROU19XUklURSIsIlZSU0xDTV9XUklURSIsIkROU19SRUFEIiwiU0VSVklDRV9BQ0NPVU5UX1JFQUQiLCJTRERDX0ZFREVSQVRJT05fUkVBRCIsIkRPTUFJTl9SRUFEIiwiVlJTTENNX1JFQUQiLCJVUEdSQURFX1dSSVRFIl0sInJvbGUiOlsiQURNSU4iXX0.w5H0W8NhbTwXafrsWJ0dNBg4WphsENoqO5ERLpGXZVQ", "refreshToken": { "id": "37f118d1-1264-4e65-b52a-9b763f2f582a" }}vcf@sddc-manager [ ~ ]$vcf@sddc-manager [ ~ ]$Set access token from above response to a variable access_token:vcf@sddc-manager [ ~ ]$ access_token=eyJhbGciOiJIUzI1NiJ9.eyJqdGkiOiI5ZGYxNzU0Yi04ZDc3LTQzODMtYTEyZC0wOTdkMzJiZTEzMGQiLCJpYXQiOjE2MjczODIyODksInN1YiI6InZjZi1zZWN1cmUtdXNlckB2c3BoZXJlLmxvY2FsIiwiaXNzIjoidmNmLWF1dGgiLCJhdWQiOiJzZGRjLXNlcnZpY2VzIiwibmJmIjoxNjI3MzgyMjg5LCJleHAiOjE2MjczODU4ODksInVzZXIiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsIm5hbWUiOiJ2Y2Ytc2VjdXJlLXVzZXJAdnNwaGVyZS5sb2NhbCIsInNjb3BlIjpbIlNERENfRkVERVJBVElPTl9XUklURSIsIkFWTl9XUklURSIsIlNERENfTUFOQUdFUl9SRUFEIiwiQ0VSVF9XUklURSIsIkNPTVBPU0FCSUxJVFlfV1JJVEUiLCJMSUNFTlNFX0tFWV9SRUFEIiwiQ09NUE9TQUJJTElUWV9SRUFEIiwiRURHRV9DTFVTVEVSX1dSSVRFIiwiVVNFUl9SRUFEIiwiQ1JFREVOVElBTF9XUklURSIsIkJBQ0tVUF9DT05GSUdfUkVBRCIsIkNMVVNURVJfV1JJVEUiLCJBVk5fUkVBRCIsIlZBU0FfUFJPVklERVJfUkVBRCIsIkRPTUFJTl9XUklURSIsIkNFSVBfUkVBRCIsIlNPU19XUklURSIsIlNERENfTUFOQUdFUl9XUklURSIsIk5UUF9XUklURSIsIkRFUE9UX0NPTkZJR19XUklURSIsIkRFUE9UX0NPTkZJR19SRUFEIiwiSE9TVF9XUklURSIsIkJBQ0tVUF9SRVNUT1JFX1JFQUQiLCJDRVJUX1JFQUQiLCJVU0VSX1dSSVRFIiwiVVBHUkFERV9SRUFEIiwiT1RIRVJfUkVBRCIsIlNPU19SRUFEIiwiU0VDVVJJVFlfQ09ORklHX1JFQUQiLCJDUkVERU5USUFMX1JFQUQiLCJIT1NUX1JFQUQiLCJDRUlQX1dSSVRFIiwiT1RIRVJfV1JJVEUiLCJMSUNFTlNFX0tFWV9XUklURSIsIkNBX1JFQUQiLCJORVRXT1JLX1BPT0xfV1JJVEUiLCJXQ1BfUkVBRCIsIkJBQ0tVUF9SRVNUT1JFX1dSSVRFIiwiTlRQX1JFQUQiLCJFREdFX0NMVVNURVJfUkVBRCIsIkJBQ0tVUF9DT05GSUdfV1JJVEUiLCJXQ1BfV1JJVEUiLCJTRVJWSUNFX0FDQ09VTlRfV1JJVEUiLCJORVRXT1JLX1BPT0xfUkVBRCIsIkNBX1dSSVRFIiwiQ0xVU1RFUl9SRUFEIiwiVkFTQV9QUk9WSURFUl9XUklURSIsIkROU19XUklURSIsIlZSU0xDTV9XUklURSIsIkROU19SRUFEIiwiU0VSVklDRV9BQ0NPVU5UX1JFQUQiLCJTRERDX0ZFREVSQVRJT05fUkVBRCIsIkRPTUFJTl9SRUFEIiwiVlJTTENNX1JFQUQiLCJVUEdSQURFX1dSSVRFIl0sInJvbGUiOlsiQURNSU4iXX0.w5H0W8NhbTwXafrsWJ0dNBg4WphsENoqO5ERLpGXZVQ On the additional VMware Cloud Foundation instance, perform a credentials lookup using REST API vcf@sddc-manager [ ~ ]$ curl localhost/v1/credentials?resourceType=PSC -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq % Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 531 0 531 0 0 2794 0 --:--:-- --:--:-- --:--:-- 2794{ "elements": [ { "id": "8491d072-c0eb-4fea-a594-7f4b9253dc97", "credentialType": "SSO", "accountType": "SYSTEM", "username": "administrator@vsphere.local", "password": "**********", -->old SSO Admin Password "creationTimestamp": "2021-07-21T22:30:59.760Z", "modificationTimestamp": "2021-07-21T22:30:59.760Z", "resource": { "resourceId": "e48b6ad3-4256-4156-a795-99b957c943b0", "resourceName": "vcenter-1.vrack.vsphere.local", "resourceIp": "x.x.x.x", --->vCenter IP "resourceType": "PSC", "domainName": "sddcId-1001" } } ], "pageMetadata": { "pageNumber": 0, "pageSize": 1, "totalElements": 1, "totalPages": 1 }}vcf@sddc-manager [ ~ ]$ On the additional VMware Cloud Foundation instance, perform a REMEDIATE operation, providing the json data from the result of step 3, but the new password from primary instance. The response will have task Id. vcf@sddc-manager [ ~ ]$ cat remediate.json{ "operationType" : "REMEDIATE", "elements" : [ { "resourceName" : "vcenter-1.vrack.vsphere.local", "resourceType" : "PSC", "credentials" : [ { "username" : "administrator@vsphere.local", "password" : "***********" --> New Password } ] } ]}vcf@sddc-manager [ ~ ]$ curl localhost/v1/credentials -X PATCH -d@remediate.json -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 315 0 68 100 247 84 305 --:--:-- --:--:-- --:--:-- 389{ "id": "bbd90419-99cc-4677-a5dd-927af03d3bab", "status": "IN_PROGRESS"} On the additional VMware Cloud Foundation instance, monitor task status by using the task id. Please refer to the VMware Cloud Foundation Rest API documentation. *Once the status is successful, VCF secondary instance is now synced and SDDC Manager UI should start working. vcf@sddc-manager [ ~ ]$ curl localhost/v1/tasks/bbd90419-99cc-4677-a5dd-927af03d3bab -H "Authorization: Bearer $access_token" -H "Content-Type: application/json" | jq% Total % Received % Xferd Average Speed Time Time Time Current Dload Upload Total Spent Left Speed100 841 0 841 0 0 17893 0 --:--:-- --:--:-- --:--:-- 17893{ "id": "bbd90419-99cc-4677-a5dd-927af03d3bab", "name": "Credentials remediate operation", "status": "SUCCESSFUL", "creationTimestamp": "2021-07-27T10:38:39.007Z", "subTasks": [ { "name": "Password remediate prevalidation", "description": "Prevalidation of password remediate request", "status": "SUCCESSFUL", "creationTimestamp": "2021-07-27T10:38:39.007Z", "completionTimestamp": "2021-07-27T10:38:39.007Z" }, { "name": "Password remediate for resource : vcenter-1.vrack.vsphere.local, user : administrator@vsphere.local and credential type : SSO", "description": "Password remediate for resource : vcenter-1.vrack.vsphere.local, user : administrator@vsphere.local and credential type : SSO", "status": "SUCCESSFUL", "creationTimestamp": "2021-07-27T10:38:40.090Z", "completionTimestamp": "2021-07-27T10:38:40.090Z" } ], "resolutionStatus": "UNRESOLVED", "isCancellable": false}