Symptoms

Changes made to the /etc/ntp.conf file or other NTP related changes, possible NTP server failure, possible network change that causes the NTP server(s) to be unreachable The VxRail upgrade from 4.5.310 to 4.7.410 failed with the below error in the lcm.log but it does not show the actual issue, the actual issue is found in the web.log further below, this is not a version specific issue/var/log/mystic/lcm.log 2020-04-04T14:23:37.884Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.log2020-04-04T14:23:37.884Z INFO root Exiting with exit-code 12020-04-04T14:23:44.576+0000 INFO [pool-4-thread-1] com.vce.lcm.service.upgrade.vc.task.VCVAStartUpgradeTask VCVAStartUpgradeTask.downloadLogsForDignosis:321 - ==================================================2020-04-04T14:23:44.576+0000 INFO [pool-4-thread-1] com.vce.commons.core.services.VCConnectionServiceImpl VCConnectionServiceImpl.getVCConnection:92 - Time before get VCConnection Sat Apr 04 14:23:44 UTC 20202020-04-04T14:23:44.584+0000 INFO [pool-4-thread-1] com.vce.commons.core.services.VCConnectionServiceImpl VCConnectionServiceImpl.getVCConnection:94 - Time after get VCConnection Sat Apr 04 14:23:44 UTC 20202020-04-04T14:23:44.585+0000 INFO [pool-4-thread-1] com.vce.commons.core.services.VCConnectionServiceImpl VCConnectionServiceImpl.getVCConnection:98 - This is to identify how many vcConnection in the connectionMap ======2020-04-04T14:23:44.585+0000 INFO [pool-4-thread-1] com.vce.commons.core.services.VCConnectionServiceImpl VCConnectionServiceImpl.getVCConnection:99 - ConnectionMap size is 12020-04-04T14:23:44.585+0000 INFO [pool-4-thread-1] com.vce.commons.core.services.VCConnectionServiceImpl VCConnectionServiceImpl.lambda$getVCConnection$0:101 - key administrator@vsphere.local, instance com.vce.commons.core.connection.VCConnection@434dddc12020-04-04T14:23:45.597+0000 ERROR [pool-6-thread-1] com.vce.lcm.task.SimpleUpgradeTaskExecutor SimpleUpgradeTaskExecutor.execute:67 - Failed to execute task PscStartUpgradeTask.java.util.concurrent.ExecutionException: com.vce.lcm.exception.LCMInternalException: Failed to pre-check virtual machine VMware vCenter Server Platform Services Controller (QDamAcK) meets upgrade requirements. at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[?:1.8.0_231] at java.util.concurrent.FutureTask.get(FutureTask.java:206) ~[?:1.8.0_231] at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:57) [lcm_module-4.7.410.jar:?] at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:105) [lcm_module-4.7.410.jar:?] at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:111) [lcm_module-4.7.410.jar:?] at com.vce.lcm.task.SimpleUpgradeTaskExecutor.execute(SimpleUpgradeTaskExecutor.java:128) [lcm_module-4.7.410.jar:?]..............................Caused by: com.vce.lcm.exception.LCMInternalException: Failed to pre-check virtual machine VMware vCenter Server Platform Services Controller (QDamAcK) meets upgrade requirements. at com.vce.lcm.service.upgrade.vc.task.VCVAStartUpgradeTask.notifyActionProgress(VCVAStartUpgradeTask.java:302) ~[lcm_module-4.7.410.jar:?] at com.vce.lcm.service.upgrade.vc.task.VCVAStartUpgradeTask.updateProgress(VCVAStartUpgradeTask.java:246) ~[lcm_module-4.7.410.jar:?].............................. /var/log/mystic/web.log 2020-04-04T17:45:35.981Z INFO base_commands Executing command: []2020-04-04T17:45:35.985Z INFO featureState_utils Found FSS 'VCHA_Upgrade' with value 'True'2020-04-04T17:45:36.319Z INFO os_utils Trying to retrieve the host 10.13.53.101 python3 path2020-04-04T17:46:10.573Z ERROR transport Command ['/usr/bin/python3', '/tmp/vmware-upgrade-temp-dirF1t4Jifr4q/tmpaCzPLgIJcL/bootstrap_scripts/run_linux_preupgrade_checks.py', '--inputFile', 'upgrade-requirements-config.json', '--outputFile', 'upgrade-requirements-output.json', '--logDir', '/var/log/vmware/upgrade', '--logFileName', 'upgrade-source-requirements.log', '--disableScreenLog', '--logLevel', 'INFO'] exit-code=1, stdout=, stderr=2020-04-04T17:46:10.574Z ERROR upgrade_commands Pre-upgrade checks failed. Check upgrade-source-requirements.log log for details.2020-04-04T17:46:10.791Z INFO upgrade_commands Reporting source preupgrade result errors and warnings.2020-04-04T17:46:10.791Z ERROR upgrade_commands Pre-upgrade checks errors:2020-04-04T17:46:10.791Z ERROR upgrade_commands {'problemId': None, 'text': {'translatable': 'The source vCenter Server Appliance cannot access the configured NTP servers: %(0)s or there is difference between the local time and the timereturned from the NTP servers', 'args': ['10.34.10.20'], 'localized': 'The source vCenter Server Appliance cannot access the configured NTP servers: 10.34.10.20 or there is difference between the local time and the time returned from the NTP servers', 'id': 'ur.precheck.timesync.error.text'}, 'resolution': {'translatable': 'You can check the appliance NTP configuration in the /etc/ntp.conf file. Make sure that the local time of the appliance is synchronized with the time from the NTP servers. Make sure that there are no connectivity issues between the appliance and the NTP servers', 'localized': 'You can check the appliance NTP configuration in the /etc/ntp.conf file. Make sure that the local time of the appliance is synchronized with the time from the NTP servers. Make sure that there are no connectivity issues between the appliance and the NTP servers', 'id': 'ur.precheck.timesync.error.resolution'}, 'description': {'translatable':'The source appliance cannot access the configured NTP servers: %(0)s or there is difference between the local time and the time returned from the NTP servers', 'args': ['10.34.10.20'], 'localized': 'The source appliance cannot access the configured NTP servers: 10.34.10.20 or there is difference between the local time and the time returned from the NTP servers', 'id': 'ur.precheck.timesync.error.description'}}2020-04-04T17:46:10.791Z ERROR upgrade_commands Pre-upgrade checks failed on source host. Check upgrade-source-requirements.log log for details.

Cause

This can be caused either by a time skew between the service vm's or that any one of the service vm's is not able to reach its configured ntp server. We also found in this case that the PSC and VXRM were using a different NTP server from vCenter. root@VCSA [ ~ ]# cat /etc/ntp.conftinker panic 0restrict default kod nomodify notrap nopeerrestrict 127.0.0.1restrict -6 ::1driftfile /var/lib/ntp/drift/ntp.driftserver 10.13.53.1root@vPSC [ /bin ]# cat /etc/ntp.conftinker panic 0restrict default kod nomodify notrap nopeerrestrict 127.0.0.1restrict -6 ::1driftfile /var/lib/ntp/drift/ntp.driftserver 10.34.10.20VXRM:~ # cat /etc/ntp.conftinker panic 0server 10.34.10.20restrict -4 default kod notrap nomodify nopeer noqueryrestrict -6 default kod notrap nomodify nopeer noqueryrestrict 127.0.0.1restrict ::1driftfile /var/lib/ntp/drift/ntp.driftlogfile /var/log/ntpNo time skew exists but when running # ntpq -p on the PSC vm we found it could not reach its configured NTP server

Resolution

Confirm whether or not there is a time skew between the service vm's and resolve it if there is. The service vm's are: vCenter Server Appliance, Platform Service Controller, and VxRail Manager Run the 'date' command on each service vm and make sure they are the sameIf they are different then resolve the time skew If there is no time skew, check the /etc/ntp.conf of each service vm to verify if they are using the correct NTP servers which will have to be verified by the customer Run # cat /etc/ntp.conf to print the ntp configuration of each service vm root@service_vm [ ~ ]# cat /etc/ntp.conftinker panic 0restrict default kod nomodify notrap nopeerrestrict 127.0.0.1restrict -6 ::1driftfile /var/lib/ntp/drift/ntp.driftserver 10.13.53.1 - check with customer to make sure this ip address is the correct address for the NTP server for each service vm If you can confirm this is correct, use the ntpq -p command to check if the vm is able to reach its configured NTP server, in this case the PSC vm was not able to reach its configured NTP serverExample of bad ntpq -p output, pay attention to the 'reach' column, this gives you the status of the last eight NTP messages and should have a value of 377 root@psc-1 [ ~ ]# ntpq -p remote refid st t when poll reach delay offset jitter============================================================================== ntp_server01 .INIT. 16 u - 64 0 0.000 0.000 0.000 ntp_server02 .INIT. 16 u - 64 0 0.000 0.000 0.000 Example of good ntpq -p ouput: root@psc-1 [ ~ ]# ntpq -p remote refid st t when poll reach delay offset jitter==============================================================================ntp_server01 10.254.140.21 3 u 6 128 377 0.132 1.341 0.275ntp_server02 10.254.140.21 3 u 32 128 377 0.115 1.432 0.290 In this case the PSC could not reach its intended NTP server, so we updated all the service vm's to have the same NTP server as the VCSA vm which was known to be working. You can update the NTP servers being used by modifying the /etc/ntp.conf and then restarting services or use the VAMI page to update the NTP in the gui.After making this change and restarting the upgrade it should run through without any issues.

Support Cases