Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
The components of the vSphere environment are not time synchronized.Firstboot fails during Install/Deployment, Upgrade or Migration.In the firstbootStatus.json file, you may see one of these services have failed to configure or start during firstboot: "failedSteps": "cmfirstboot""failedSteps": "analytics_firstboot""failedSteps": "vpxd_firstboot""failedSteps": "pschealth-firstboot”"failedSteps": "sms_spbm_firstboot""failedSteps": "vmafd-firstboot""failedSteps": "vapi_firstboot""failedSteps": "mgmt-firstboot""failedSteps": "scafirstboot""failedSteps": "updatemgr-firstboot""failedSteps": "ngc_firstboot" In the cmfirstboot.py_####_stderr.log file, you see the error: PAM: Authentication token is no longer valid An error occurred while performing security operation: 'Failed to add user: cm to group: cis’ In the analytics_firstboot.py_####_stderr.log file, you see the error: Analytics Service registration with Component Manager failed ns0:MessageExpired The time now (date + time) does not fall in the request lifetime interval extended with clock tolerance of 600000 ms [ (date + time); (date +time)]. This might be due to a clock skew problem. In the vpxd_firstboot.py_####_stdout.log file, you see the error: ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720) In the pschealth-firstboot.py_#####_stderr.log file, you see the error: An error occurred while starting service 'pschealth' In the sms_spbm_firstboot.py_####_stderr.log file, you see the error: VMware vSphere Profile-Driven Storage Service failed to start In the vmafdd-syslog.log file, you see the error: Vmdir server is down. In the vmdird-syslog.log file, you see the error: DecodeEntry failed (9605) DN:() LoadServerGlobals: (9700)() In the vapi_firstboot.py_####_stderr.log file, you see the error: Failed to configure vAPI Endpoint Service at the firstboot time In the mgmt-firstboot.py_####_stderr.log file, you see the error: UnboundLocalError: local variable 'e' referenced before assignment In the scafirstboot.py_####_stderr.log file, you see the error: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720) In the updatemgr-firstboot.py_6012_stderr.log file, you see the error: Failed to register updatemgr extension In the ngc_firstboot.py_####_stderr.log file, you see the error: SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:720) In the %ProgramData%\VMware\vCenterServer\logs\cm\cm.log file, you see entries similar to: Caused by: com.vmware.vim.vmomi.client.exception.VlsiCertificateException: Server certificate chain is not trusted and thumbprint verification is not configuredat com.vmware.vim.vmomi.client.http.impl.ThumbprintTrustManager.checkServerTrusted(ThumbprintTrustManager.java:183)at sun.security.ssl.AbstractTrustManagerWrapper.checkServerTrusted(SSLContextImpl.java:984)at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1496)... 78 moreCaused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: timestamp check failed In the fbInstall.json file shows an unexpected time discrepancy between the start time and end time. This could be a large jump forward or backward in time. For example: "start_time": "2018-05-07T13:00:00.000Z"end_time": "2018-05-07T18:00:00.000Z"or"start_time": "2018-05-07T18:05:00.000Z"end_time": "2018-05-07T18:00:00.000Z"
This issue occurs due to time inconsistencies in the vSphere environment. The issue most commonly happens when the target ESXi host for the destination vCenter Server Appliance is not synchronized with NTP. This issue can also happen if the destination vCenter Server Appliance migrates to an ESXi host with different time due to fully automated DRS.
To avoid time synchronization issues, ensure the following is correct before deploying, migrating, or upgrading a vCenter Server Appliance: The target ESXi host where the destination vCenter Server Appliance will be deployed is synchronized to NTP.If the target ESXi host is part of a Fully Automated DRS cluster, change the automation level to Manual.The ESXi host running the source vCenter Server Appliance is synchronized to NTP.If the vCenter Server Appliance will be connected to an external Platform Services Controller, ensure the ESXi host running the external Platform Services Controller is synchronized to NTP.Verify that the source vCenter Server or vCenter Server Appliance and external Platform Services Controller have the correct time. For more information on: Managing time in vSphere, see Synchronizing Clocks on the vSphere Network.Changing DRS Automation, see Edit Cluster Settings.vCenter Server Appliance requirements, see System Requirements for the vCenter Server Appliance and Platform Services Controller Appliance.
Note: Firstboot logs are located at: vCenter Server Appliance - Firstboot logs are located in the /var/log/firstboot directory.vCenter Server on Windows - Firstboot logs are located in the VMware-VCS-logs-.zip/vcs_logs/uninstall directory or VMware-VCS-logs-/vcs_logs/uninstall/.zip/ProgramData/VMware/vCenterServer/logs/firstboot directory Note: In vSphere 7.0, vCenter Server for Windows has been removed and support is not available. For more information, see Farewell, vCenter Server for Windows . To collect a log bundle or review log files: Triaging a vCenter Server Appliance 6.0 installation, upgrade, or migration (6.0 U2m) failure (2106760)Triaging a vCenter Server 6.0 installation or upgrade failure (2105258) For more information, see VMware vCenter Failed Firstboot.-------------------------------------------------------------------------------------------------------------------------------------------------For more information see: Error - An error occurred while starting service 'pschealth' Synchronizing ESXi/ESX time with a Microsoft Domain Controller"Newly deployed vCenter Server Appliance time is not synchronized with Platform Services Controller" while deploying vCenter from 6.x to 6.7 fails at Pre-Migration check