Symptoms

Scenario 1: The vCenter certificate is already expired. (For all VxRail versions) Unable to log in to vCenter UI.Any log-in attempt when the Web UI is available fails even with correct credentials. Restart of vCenter Server Appliance (VCSA) services fails.Restart of services does not bring up all services. Errors observed: /var/log/vmware/vpxd-svcs/vpxd-svcs.log: 2020-06-03T09:31:04.523Z [pool-8-thread-1 INFO com.vmware.identity.token.impl.X509TrustChainKeySelector opId=905f6864-c067-4db6-828c-1d59c4b43bf8] Failed to find trusted path to signing certificate sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target at sun.security.provider.certpath.SunCertPathBuilder.build(SunCertPathBuilder.java:141) Scenario 2: The vCenter certificate expires in less than 60 days. (For VxRail 7.0.480 and above versions) Log in to vCenter UI is completed but VxRail 7.0.480 and later versions show a Warning in VxRail Cluster > Configure > VxRail > Certificate > All Trust Store Certificates page stating that the certificate expires in less than 60 days.

Cause

vCenter certificates are expired or expire soon.VxRail versions which were initially built prior to 4.7, may have certificates issued with a lifespan of two years from the date of installation. At the time of writing of this article, a VxRail build on 4.7.410 has all certificates with a 10-year lifespan.Minor version upgrades do not touch the certificates!For a VxRail which was initially built on 4.5.210 and later versions, the certificates have a two-year validity period. Check the VMware article for VMware Security Token Service (STS) Checking Expiration of STS Certificate on vCenter Servers (79248) to confirm the detailed description.Use the view the certificate in the browser of the log-in page of the VCSA to confirm the certificate has expired. Or list the certificates in the CLI of the Platform Services Controller (PSC) (VCSA). See commands from VMware article Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719) for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done

Resolution

For Scenario 1, when the vCenter certificate is already expired, follow the below procedure to generate new self-signed certificates on PSC and VCSA. Note: This procedure is intended for single PSC or VCSA VMs which are maintained through VxRail LifeCycle Manager (LCM). For HA, ELM, or Customer deployed VCSAs, open a VMware ticket! Note: Take OFFLINE snapshots of VxRail Manager (VRM), PSC, and VCSA! Note: Check if the snapshot creating process has finished without errors! Do NOT continue without valid snapshots! Note: If issues are encountered, do not retry without reverting to snapshots! Fix PSC: Reset all Certificates (This fails but that is expected.) Start Certificate Manager: /usr/lib/vmware-vmca/bin/certificate-manager Select Option 8 > Reset all Certificates Confirm "Do you wish to generate all certificates using configuration file : Option[Y/N] ?" Enter Credentials Enter Values Leave the "IPAddress" field emptyEnter Hostname as Fully Qualified Domain Name (FQDN) of PSCVMware Certificate Authority (VMCA) Name field is the name of a new root CA being created, for example, VxRail CA. Confirm "Continue operation : Option[Y/N] ?" Confirm "Continue operation : Option[Y/N] ?" This operation fails with: Get site nameCompleted [Reset Machine SSL Cert...] g3node-site Lookup all services Get service xxxxxx-site:xxxxxxx-d202-4d8f-9282-xxxxxx317b8f Update service xxxxxx-site:xxxxxxxx-d202-4d8f-9282-xxxxxx317b8f; spec: /tmp/svcspec_a1hipoqq Status : 0% Completed [Reset operation failed] please see /var/log/vmware/vmcad/certificate-manager.log for more information. root@xxxxc [ ~ ]# Fix the STS issue Download and run the script from VMware article "Signing certificate is not valid" error in VCSA 6.5.x,6.7.x or vCenter Server 7.0.x, 8.0.x. (76719) Stop Services service-control --all --stop Start Services (This fails but that is expected) service-control --all --start Wait for the process to time out or stop it when it gets to the "vmware-vmon" service /usr/lib/vmware-vmca/bin/certificate-manager Select Option 6 > "Replace Solution user certificates with VMCA certificates"Confirm "Do you wish to generate all certificates using configuration file : Option[Y/N] ?" Enter CredentialsDeny (enter "N") for reconfigure as all options were configured above "certool.cfg file exists, Do you wish to reconfigure : Option[Y/N] ?" Confirm "You are going to regenerate Solution User Certificates using VMCA, Continue operation : Option[Y/N] ?" Wait until the procedure exits. This procedure: Generates all certificatesStops the servicesStarts the services Confirm if all Services are running service-control --all --status Fix Certificates on VCSA Stop and start all services. This MUST be done AFTER all PSC services are running! Stop service-control --all --stop Start service-control --all --start Wait for the process to time out or stop it when it gets to the vmware-vmon service /usr/lib/vmware-vmca/bin/certificate-manager Select Option 8 > Reset all CertificatesStart Certificate ManagerConfirm "Do you wish to generate all certificates using configuration file : Option[Y/N] ?" Enter Credentials Enter PSC IPEnter Values Leave IPAddress field emptyEnter Hostname as FQDN of VCSAThe VMCA Name field is the name of the new root CA being created, for example, VxRail CA. Confirm "Continue operation : Option[Y/N] ?" Confirm "Continue operation : Option[Y/N] ?" Wait until all certificates are generated and a successful completion message appears "Reset status : 100% Completed [Reset completed successfully]" Check that all services are running service-control --all --status Access vCenter UIAccess by Domain Name System (DNS) entry fails in Chrome due to HTTP Strict Transport Security (HSTS). Open the VCSA IP or use another supported browser such as FireFox. For Scenario 2, when the vCenter certificate expires in less than 60 days, follow the below procedure to renew the certificate in advance to avoid VxRail manager disconnect from vCenter. Log in to the vCenter over SSH as the root user Restart Services Run Stop "service-control --stop --all" Run Start "service-control --start --all" Reset all Certificates Run: "/usr/lib/vmware-vmca/bin/certificate-manager" Select Option 8 > Reset all Certificates Enter vSphere username and password Input the Certificate Properties Confirm the operation and then the vCenter root or machine Certificates are renewed Follow the article Dell VxRail: How to manually import vCenter SSL certificate on VxRail Manager to import the updated vCenter and CA certificates into the VxRail Manager trust store.

Support Cases