Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
After vCenter Server Appliance (VCSA) upgrade to 7.0 Update 1 or later, vSphere Cluster Service (vCLS) virtual machine(s) are not getting deployed.You see a warning message in vSphere Client as "vSphere DRS functionality was impacted due to unhealthy state vSphere Cluster Services caused by the unavailability of vSphere Cluster Service VMs. vSphere Cluster Service VMs are required to maintain the health of vSphere DRS"You may see the below errors/warnings:Can’t provision VM for Cluster Agent due to lack of suitable datastoreCouldn’t acquire token due to: Signature validation failedYou may see the below error snippets in /var/log/vmware/eam/eam.log 2020-11-17T10:57:19.312Z | INFO | cluster-agent-1 | AgentBase.java | 229 | [checkGoal:ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null')] task in progress.2020-11-17T10:57:19.312Z | INFO | cluster-agent-1 | VcEventManager.java | 792 | [EventIndex: 141046] Posting event.2020-11-17T10:57:19.312Z | ERROR | cluster-agent-1 | AuditedJob.java | 106 | JOB FAILED: [#1878814658] DeployVmJob(ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null'))com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null') due to lack of suitable datastore.2020-11-17T10:57:19.312Z | INFO | cluster-agent-1 | AgentBase.java | 229 | [checkGoal:ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null')] task in progress.2020-11-17T10:57:19.312Z | INFO | cluster-agent-1 | VcEventManager.java | 792 | [EventIndex: 141046] Posting event.2020-11-17T10:57:19.312Z | ERROR | cluster-agent-1 | AuditedJob.java | 106 | JOB FAILED: [#1878814658] DeployVmJob(ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null'))com.vmware.eam.job.DeployVmJob$DeployVmJobFailure: Can't provision VM for ClusterAgent(ID: 'Agent:5a001f25-7de6-483a-8c0f-1eb5199515dd:null') due to lack of suitable datastore. 2020-11-17T10:57:49.103Z | INFO | sts-0 | Workflow.java | 121 | [CreateSAMLToken:577f77cb515aed10] FAILEDcom.vmware.eam.sso.exception.TokenNotAcquired: Couldn't acquire token due to: Signature validation failedCaused by: com.vmware.vim.sso.client.exception.MalformedTokenException: Signature validation failedNote:The preceding log excerpts are only examples. Date,time and environmental variables may vary depending on your environment
As part of the vCLS deployment workflow, EAM Service will identify the suitable datastore to place the vCLS VMs. This workflow was failing due to EAM Service unable to validate the STS Certificate in the token.
Warning: This script interacts with the VMDIR's database. Take an offline snapshot concurrently for all vCenter Servers in the SSO domain before running the script. Failing to do so may result in an unrecoverable error and require redeploying vCenter Server.Once the script is complete, restart services for all vCenters in the site domain. As such, the below script fix will require outages for all vCenters in the site domain.
This issue is resolved in VMware vCenter Server 7.0 Update 3, available at VMware Downloads.
To workaround this issue, reset the STS Certificate using the attached fixsts script, detailed procedure mentioned below :Note: This script should only be run once per SSO domain; STS certs are thereby updated for all vCenters in the SSO domain. Please follow the below steps: Download the attached fixsts.sh script from this article and upload to the /tmp folder on impacted vCenter Server.If the connection to upload to the vCenter by the SCP client is rejected, run this from an SSH session to the vCenter: chsh -s /bin/bash, refer to Error when uploading files to vCenter Server Appliance using WinSCP for more information.Connect to the vCenter Server with an SSH session if you have not already per Step 2.Navigate to the /tmp directory ( cd /tmp ) :Run chmod +x fixsts.sh to make the file executable.Run ./fixsts.sh.Restart services on all vCenters in your SSO domain by using below commands: service-control --stop --all service-control --start --allvCLS VM(s) should get deployed successfully
"Signing certificate is not valid" error in VCSA 6.5.x/6.7.x and vCenter Server 7.0.xFor more information on vCLS, see vSphere Cluster Services (vCLS) in vSphere 7.0 Update 1For more information on STS certificates, see Security Token Service STS