OPERATIONAL DEFECT DATABASE
...
...
When the vCenter Server SSL Certificates expire, you experience these symptoms: Cannot log in to vCenter Server using the vSphere Web Client Logging in to vCenter Server using the vSphere Web Client fails You see this error: Cannot connect to vCenter Single Sign On server https://vc.domain.com:7444/ims/STSService?wsdl. The SSL certificate cannot be verified. The VMware VirtualCenter Server service cannot start. In the vpxd.log file, located at /var/log/vmware/vpx/vpxd.log, you see entries similar to: <YYYY-DD-MM>< TIME> [03992 error 'HttpConnectionPool-000001'] [ConnectComplete] Connect failed to <cs p:0000000008165ed0, TCP:vc.domain.com:7444>; cnx: (null), error: class Vmacore::Ssl::SSLVerifyException(SSL Exception: Verification parameters: --> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF --> ExpectedThumbprint: --> ExpectedPeerName: vc.domain.com --> The remote host certificate has these problems: --> --> * certificate has expired) <YYYY-DD-MM>< TIME> [03884 error '[SSO][SsoFactory_CreateFacade]'] Unable to create SSO facade: SSL Exception: Verification parameters: --> PeerThumbprint: 39:D4:04:4A:FB:AC:8E:05:EC:45:22:81:3F:45:28:44:4C:C7:25:DF --> ExpectedThumbprint: --> ExpectedPeerName: vc.domain.com --> The remote host certificate has these problems: --> --> * certificate has expired. <YYYY-DD-MM>< TIME> [03884 error 'vpxdvpxdMain'] [Vpxd::ServerApp::Init] Init failed: Vpx::Common::Sso::SsoFactory_CreateFacade(sslContext, ssoFacadeConstPtr) --> Backtrace: --> backtrace[00] rip 000000018018b86a --> backtrace[01] rip 0000000180102ac8 --> backtrace[02] rip 0000000180103f9e --> backtrace[03] rip 000000018008d22b --> backtrace[04] rip 00000000004e5bdc --> backtrace[05] rip 0000000000506652 --> backtrace[06] rip 00007ff71e14f001 --> backtrace[07] rip 00007ff71e148e1c --> backtrace[08] rip 00007ff71e36d8db --> backtrace[09] rip 00007ffe927381d5 --> backtrace[10] rip 00007ffe927b16ad --> backtrace[11] rip 00007ffe92a94409 --> <YYYY-DD-MM>< TIME> [03884 warning 'VpxProfiler'] ServerApp::Init [TotalTime] took 5015 ms <YYYY-DD-MM>< TIME> [03884 error 'Default'] Failed to intialize VMware VirtualCenter. Shutting down... <YYYY-DD-MM>< TIME> [03884 info 'vpxdvpxdSupportManager'] Wrote uptime information
Notes: This article assumes that you have prepared new and valid SSL Certificates. If you have not yet prepared new certificates, then see the Generating the certificate requests and Getting the certificates sections of the Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (2057223). This article uses examples based on the new certificates being stored in these directories: /ssl/vpxd/ /ssl/inventory/ /ssl/logbrowser/ /ssl/autodeploy/ Before continuing, ensure that you have a valid working backup of the vCenter Server Appliance. Take a snapshot of the vCenter Server Appliance virtual machine before proceeding. To resolve this issue, recover the expired SSL certificates in VMware vCenter Server Appliance 5.5.x:To recover the expired SSL certificates: Run this command to back up the vpxd.cfg file:cp /etc/vmware-vpx/vpxd.cfg /etc/vmware-vpx/vpxd.cfg.backup Edit the /etc/vmware-vpx/vpxd.cfg file using vi editor.To edit the vpxd.cfg file: Open the vi /etc/vmware-vpx/vpxd.cfg file using vi editor. Locate this line:<enabled>true</enabled> Press i to change to INSERT mode. Change this line to:<enabled>false<enabled> Press ESC :wq ENTER to save the file and quit. Replace the Single Sign-on and VMware VirtualCenter Server service certificate:To replace the Single Sign-on and VMware VirtualCenter Server service certificate: Stop the VMware VirtualCenter Server service and the vCenter Single Sign-On service using these commands:service vmware-stsd stopservice vmware-vpxd stop Create the chain.pem file for VMware VirtualCenter Server service by running this command:cat /ssl/vpxd/rui.crt /ssl/vpxd/cachain.pem > /ssl/vpxd/chain.pem Replace the SSL certificate by running this command:/usr/sbin/vpxd_servicecfg certificate change /ssl/vpxd/chain.pem /ssl/vpxd/rui.keyWait until you receive this response:VC_CFG_RESULT = 0 Revert the changes to the vpxd.cfg file performed in Step 2.To edit the vpxd.cfg file: Open the vi /etc/vmware-vpx/vpxd.cfg file using vi editor. Locate this line:<enabled>false</enabled> Press i to change to INSERT mode. Change this line to:<enabled>true<enabled> Press ESC :wq ENTER to save the file and quit. Run this command to un-register the VMware VirtualCenter Server serviceID from Single Sign-On:/usr/lib/vmware-sso/bin/vi_regtool unregisterService -d https://vcenter_FQDN:7444/lookupservice/sdk -u administrator@vsphere.local -p PASSWORD -si /etc/vmware-vpx/ls-service-id Run this command to check and note the vCenter Server Solution User:grep name /etc/vmware-vpx/vpxd.cfg<name>vpxd-vcva55.domain.com-7032d8b5-63a2-4e8d-b025-a07f8852a75f</name> Run this command to un-register the vCenter Server Solution User from Single Sign-On:/usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u administrator@vsphere.local -p PASSWORD -su vpxd-vcva55.domain.com-e8b409a5-40da-4353-8546-48eaf7608045 Run this command to re-register vCenter Server with Single Sign-On:/etc/vmware-sso/register-hooks.d/01-vcenter --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user administrator@vsphere.local --password PASSWORD --option sso-deployment-type=embedded --option ls-certificate-thumbprint=null --option vc-admin-principal=root --option vc-admin-is-group=false Run this command to check and note the Inventory Service Solution User:grep dataservice.sso.solutionUser /usr/lib/vmware-vpx/inventoryservice/lib/server/config/dataservice.propertiesdataservice.sso.solutionUser=inventory-service-162e5b30-59e9-4f9d-82ac-2718e186287f Run this command to un-register the Inventory Service Solution User:/usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u administrator@vsphere.local -p PASSWORD -su inventory-service-162e5b30-59e9-4f9d-82ac-2718e186287f Replace the Inventory Service certificate.To replace the Inventory Service certificate: Create the chain.pem file for the Inventory Service by running this command:cat /ssl/inventory/rui.crt /ssl/inventory/cachain.pem > /ssl/inventory/chain.pem Create the *.pfx file by running this command:openssl pkcs12 -export -out /ssl/inventory/rui.pfx -in /ssl/inventory/chain.pem -inkey /ssl/inventory/rui.key -name rui -passout pass:testpassword Run this command to copy the rui.key, rui.crt, and rui.pfx files to the /usr/lib/vmware-vpx/inventoryservice/ssl directory:cp /ssl/inventory/rui.* /usr/lib/vmware-vpx/inventoryservice/ssl/. Change the permissions on these files by running these commands:chmod 400 /usr/lib/vmware-vpx/inventoryservice/ssl/rui.key /usr/lib/vmware-vpx/inventoryservice/ssl/rui.pfxchmod 644 /usr/lib/vmware-vpx/inventoryservice/ssl/rui.crt Run this command to re-register the Inventory Service with Single Sign-On:/etc/vmware-sso/register-hooks.d/02-inventoryservice --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user administrator@vsphere.local --password PASSWORD Run this command to force a re-register of the Inventory Service with the vCenter Server on next restart of the VMware VirtualCenter Server service:rm /var/vmware/vpxd/inventoryservice_registered Run this command to restart the VMware VirtualCenter Server service:service vmware-vpxd restart Run this command to list the services registered to Single Sign-On and check and note the Log Browser ServiceID and Solution User (ownerID): /usr/lib/vmware-sso/bin/vi_regtool listServices https://vcenter_FQDN:7444/lookupservice/sdk Service 6 ----------- serviceId=local:b23d652b-eed8-4737-9638-2867abd9fd0a serviceName=VMware Log Browser type=urn:logbrowser:logbrowser endpoints={[url=https://vcva55.domain.com:12443/vmwb/logbrowser,protocol=unknown],[url=https://vcva55.domain.com:12443/authentication/authtoken,protocol=unknown]} version=1.0.2175565 description=Enables browsing vSphere log files within the VMware Web Client ownerId=logbrowser-vcva55.domain.com-23bc85a0-894c-435b-a3e8-19be1e371e4c productId= viSite=local Return code is: Success Run this command to create a ServiceID File for the Log Browser service:echo local:b23d652b-eed8-4737-9638-2867abd9fd0a >> /tmp/logbrowser_id Run this command to un-register the Log Browser ServiceID from Single Sign-On:/usr/lib/vmware-sso/bin/vi_regtool unregisterService -d https://vcenter_FQDN:7444/lookupservice/sdk -u administrator@vsphere.local -p PASSWORD -si /tmp/logbrowser_id Run this command to un-register the Log Browser Solution User from Single Sign-On:/usr/lib/vmware-sso/bin/vi_regtool unregisterSolution -d https://vcenter_FQDN:7444/lookupservice/sdk -u administrator@vsphere.local -p PASSWORD -su logbrowser-vcva55.domain.com-23bc85a0-894c-435b-a3e8-19be1e371e4c Replace the Log Browser certificate.To replace the Log Browser certificate: Create the chain.pem file for VMware Log Browser Service by running this command:cat /ssl/logbrowser/rui.crt /ssl/logbrowser/cachain.pem > /ssl/logbrowser/chain.pem Create the *.pfx file by running this command:openssl pkcs12 -export -out /ssl/logbrowser/rui.pfx -in /ssl/logbrowser/chain.pem -inkey /ssl/logbrowser/rui.key -name rui -passout pass:testpassword Run this command to copy the rui.key, rui.crt, and rui.pfx files to the /usr/lib/vmware-logbrowser/conf/ directory:cp rui.* /usr/lib/vmware-logbrowser/conf/ Change the permissions on these files by running these commands:chmod 400 /usr/lib/vmware-logbrowser/conf/rui.key /usr/lib/vmware-logbrowser/conf/rui.pfxchmod 644 /usr/lib/vmware-logbrowser/conf/rui.crt Run this command to re-register the Log Browser with Single Sign-On:/etc/vmware-sso/register-hooks.d/09-vmware-logbrowser --mode install --ls-server https://vcenter_FQDN:7444/lookupservice/sdk --user administrator@vsphere.local --password PASSWORD Run this command to restart the Log Browser service:service vmware-logbrowser restart Run this command to restart the vSphere Web Client service:service vsphere-client restartNote: Steps 24 to 27 are optional and do not need to be completed unless you are using Auto Deploy. Edit the /etc/vmware-rbd/autodeploy-setup.xml file:Change <serviceAddress>127.0.0.1</serviceAddress> so that it states the FQDN of the vCenter Server Appliance <serviceAddress>vcenter_FQDN</serviceAddress> Replace the Auto Deploy certificate.To replace the Auto Deploy certificate: Run these commands to copy and rename the rui.key, rui.crt files to /etc/vmware-rbd/ssl/:cp /ssl/autodeploy/rui.crt /etc/vmware-rbd/ssl/waiter.crtcp /ssl/autodeploy/rui.key /etc/vmware-rbd/ssl/waiter.key Change the permissions on these files by running these commands:chmod 644 /etc/vmware-rbd/ssl/waiter.crtchmod 400 /etc/vmware-rbd/ssl/waiter.keychown deploy:deploy /etc/vmware-rbd/ssl/waiter.crt /etc/vmware-rbd/ssl/waiter.key Run this command to re-register the Auto Deploy Service with the VMware VirtualCenter Server service:service vmware-rbd-watchdog stoprm /var/vmware/vpxd/autodeploy_registeredservice vmware-vpxd restartNote: The autodeploy_registered file may not exist. Run this command to reboot the vCenter Server Appliance:reboot After the vCenter Server Appliance is fully booted, open a Web Browser to these URLs and verify the certificate presented:Single Sign-on - https://vcenter_FQDN:7444Inventory Service - https://vcenter_FQDN:10443vCenter Server - https://vcenter_FQDN:443Log Browser - https://vcenter_FQDN:12443vSphere Web Client - https://vcenter_FQDN:9443VAMI - https://vcenter_fqdn:5480/
Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
