Symptoms

vCenter critical Services cannot be started after using vCenter Certificate Manager to reset all SSL Certificates.After updating vCenter certificates, and checking running services, vmware-vpxd, and vmware-content-library are not running. root@vcenter [ ~ ]# service-control --status --all StartPending: vmware-vapi-endpoint vmware-vpxd-svcs Running: applmgmt lwsmd vmafdd vmware-analytics vmware-cm vmware-eam vmware-postgres-archiver vmware-rhttpproxy vmware-statsmonitor vmware-vmon vmware-vpostgres vsphere-client vsphere-ui Stopped: vmcam vmonapi vmware-certificatemanagement vmware-content-library vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-perfcharts vmware-pod vmware-rbd-watchdog vmware-sca vmware-sps vmware-topologysvc vmware-updatemgr vmware-vcha vmware-vpxd vmware-vsan-health vmware-vsm vsan-dps Trying to start vmware-vpxd service you get an error similar to: root@vcenter [ ~ ]# service-control --start vmware-vpxd Operation not cancellable. Please wait for it to finish... Performing start operation on service vpxd... Error executing start on service vpxd. Details { "componentKey": null, "detail": [ { "id": "install.ciscommon.service.failstart", "translatable": "An error occurred while starting service '%(0)s'", "args": [ "vpxd" ], "localized": "An error occurred while starting service 'vpxd'" } ], "resolution": null, "problemId": null } Service-control failed. Error: { "componentKey": null, "detail": [ { "id": "install.ciscommon.service.failstart", "translatable": "An error occurred while starting service '%(0)s'", "args": [ "vpxd" ], "localized": "An error occurred while starting service 'vpxd'" } ], "resolution": null, "problemId": null } vCenter in /var/log/vmware/vmon/vmon-syslog.log: 2021-11-08T09:39:38.271729+00:00 warning vmon ssl.CertificateError: hostname 'psc.xxxx.eg' doesn't match 'psc.xxx.xxx.eg' 2021-11-08T09:40:24.574482+00:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-rhttpproxy/rhttpproxy-vmon-apihealth.py 2021-11-08T09:40:24.574698+00:00 notice vmon Skip service health check. State STOPPED, Curr request 0 2021-11-08T09:40:24.574854+00:00 notice vmon Skip service health check. State STOPPED, Curr request 0 2021-11-08T09:40:24.574997+00:00 notice vmon Constructed command: /usr/bin/python /usr/lib/vmware-vmon/vmonApiHealthCmd.py -n vmware-vpostgres -f /dev/shm/vmware-postgres-health-status.xml 2021-11-08T09:40:24.816621+00:00 notice vmon Skip service health check. State STOPPED, Curr request 0 vCenter in /storage/log/vmware/vpxd-svcs/vpxd-svcs.log: 2021-11-08T07:37:03.716Z [Thread-9 WARN com.vmware.cis.server.util.impl.InitPoolTask opId=] Init pool encountered exception: com.vmware.cis.server.util.exception.VpxdClientException at attempt 19 2021-11-08T07:37:23.725Z [Thread-9 INFO com.vmware.vim.sso.client.impl.SiteAffinityServiceDiscovery opId=] Site affinity is disabled 2021-11-08T07:37:23.750Z [Thread-9 ERROR com.vmware.vim.sso.client.impl.SoapBindingImpl opId=] Error communicating to the remote server https://urldefense.com/v3/__https://psc.xxx.eg/sts/STSService/vsphere.local__;!!LpKI!zizBGmTnALcYXLnWIITladCn-NgryRjl_R2ufCq4LGkURpqvXMuRNpET1ze8_DkPVm9P9UE4Cw$ [psc[.]xxx[.]gov[.]eg] com.sun.xml.internal.ws.client.ClientTransportException: HTTP transport error: javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: No subject alternative DNS name matching psc.xxx.eg found.

Cause

This issue occurs when upgrading vCenter Certificates while there is a DNS issue. Two records for the same IP of PSC (Platform Service Controller) are configured on DNS Server.

Resolution

1. Check if there is a DNS record mismatch and DNS connectivity. 2. Ensure that each VM IP (vCenter, PSC, and VXM) has a matching single FQDN within the domain. 3. If there are any duplicates NS records for the same IP, then the unnecessary NS records should be deleted.Note: To check DNS Server configuration on each Redhat-based Linux VM see /etc/resolv.conf .4. Ensure that vCenter certificates are not expired by running the following command line on the vCenter VM command-line interface: root@vcenter [ ~ ]# for i in $(/usr/lib/vmware-vmafd/bin/vecs-cli store list); do echo STORE $i; sudo /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store $i --text | egrep "Alias|Not After"; done The command output should be similar to the following output: STORE MACHINE_SSL_CERT Alias : __MACHINE_CERT Not After : Oct 29 12:42:18 2031 GMT STORE TRUSTED_ROOTS Alias : 50932a13985d9c33aa386868f12ba0da57f5eaff Not After : Oct 21 21:28:25 2029 GMT Alias : 32c621e354a974e2b0c31abe17fdf4beedba113d Not After : Oct 29 11:29:49 2031 GMT Alias : 342b168f6ab63a8facde75e187f8ab0a22a857e7 Not After : Oct 29 12:42:18 2031 GMT Alias : f5e20d029d8d7e3cda69079fcddc0cb5aa4d00b3 Not After : Nov 3 10:41:37 2031 GMT STORE TRUSTED_ROOT_CRLS Alias : c4268b466f6dcefaff95ea32c85d9498819a9d9c Alias : fda5f1c260738901752aaa0bcea5758d82e42ee6 Alias : e843a42ee0a5903a099c21cabf2d8b14747adf1e Alias : f3721159c59455451478b401f80d23f996f40322 STORE machine Alias : machine Not After : Oct 29 12:42:18 2031 GMT STORE vsphere-webclient Alias : vsphere-webclient Not After : Oct 29 12:42:18 2031 GMT STORE vpxd Alias : vpxd Not After : Oct 29 12:42:18 2031 GMT STORE vpxd-extension Alias : vpxd-extension Not After : Oct 29 12:42:18 2031 GMT STORE SMS Alias : sms_self_signed Not After : Oct 27 21:47:48 2029 GMT STORE APPLMGMT_PASSWORD STORE data-encipherment Alias : data-encipherment Not After : Oct 21 21:28:25 2029 GMT STORE BACKUP_STORE 5. Should there be any expired certificates, refer to KB VxRail: Unable to log in to vCenter due to expired certificates (Customer Correctable) to update expired certificates.6. On the PSC, compare the local hostname with the name that is stored in MACHINE_SS /usr/lib/vmware-vmafd/bin/vmafd-cli get-pnid --server-name localhost Output should be similar to following: psc.xxx.eg /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store MACHINE_SS Output should be similar to following: X509v3 Subject Alternative Name: email:email@acme.com, DNS:psc.xxx.eg 7. Compare the output above. If there is a mismatch, for example DNS:psc.xxx.eg.xxx.eg that was cached on the DNS Server before editing the DNS records, then proceed with the next steps. SSH to PSC VM and initiate the certificate-manager by running the following command root@psc [ ~ ]# /usr/lib/vmware-vmca/bin/certificate-manager Use option 8 -> 8. Reset all Certificates.Follow this procedure: Confirm “Do you want to generate all certificates using configuration file: Option[Y/N] "Enter credentialsEnter valuesLeave "IPAddress" field emptyEnter FQDN of PSC into "Hostname"VMCA "Name" field is the name of the new Root CA being created (e.g. "VxRail CA")Confirm "Continue operation: Option[Y/N] ?"Confirm "Continue operation : Option[Y/N] ?" Restart all services on both PSC and vCenter service-control --stop --all service-control --start --all 8. Ensure that vCenter critical services are up and running: root@vcenter [ ~ ]# service-control --status --all Running: applmgmt lwsmd vmafdd vmonapi vmware-analytics vmware-certificatemanagement vmware-cm vmware-content-library vmware-eam vmware-perfcharts vmware-postgres-archiver vmware-rhttpproxy vmware-sca vmware-sps vmware-statsmonitor vmware-topologysvc vmware-vapi-endpoint vmware-vmon vmware-vpostgres vmware-vpxd vmware-vpxd-svcs vmware-vsan-health vmware-vsm vsphere-client vsphere-ui Stopped: vmcam vmware-imagebuilder vmware-mbcs vmware-netdumper vmware-pod vmware-rbd-watchdog vmware-updatemgr vmware-vcha vsan-dps

Support Cases