OPERATIONAL DEFECT DATABASE
...
...
Upgrading vCenter Server Appliance 6.7 to 7.0 fails with the error: Internal error occurs during pre-upgrade checks.Resolution:Send upgrade logs files to GSS for further assistance. In the /var/log/vmware/upgrade/upgrade-requirements.log, you see message similar to: 2020-02-12T05:31:10.22Z ERROR __main__ ERROR: Fatal error during upgrade REQUIREMENTS. For more details take a look at: /var/log/vmware/upgrade/requirements-upgrade-runner.logTraceback (most recent call last):File "bootstrap_scripts/run-preupgrade-checks.py", line 145, in maincommandFactory = getCommandFactory(upgradeParameters, execData, reporter)File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 4063, in getCommandFactoryfactory = _getLinuxCommandFactory(upgradeParameters, settings, reporter)File "/usr/lib/vmware/cis_upgrade_runner/bootstrap_scripts/upgrade_commands.py", line 3960, in _getLinuxCommandFactoryupgradeParameters['source-host-thumbprint'])File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py", line 1600, in createGuestOpsOperationManagervmAddress, guestUser, guestPassword, sslThumbprint)File "/usr/lib/vmware/cis_upgrade_runner/libs/sdk/transport/guestops.py", line 234, in __init__.FindByIp(None, self._vmAddress, True)File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 556, in <lambda>self.f(*(self.args + (obj,) + args), **kwargs)File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/VmomiSupport.py", line 368, in _InvokeMethodreturn self._stub.InvokeMethod(self, info, args)File "/usr/lib/vmware/cis_upgrade_runner/libs/pyVmomi.zip/pyVmomi/SoapAdapter.py", line 1481, in InvokeMethodraise obj # pylint: disable-msg=E0702pyVmomi.VmomiSupport.vmodl.fault.SystemError: (vmodl.fault.SystemError) {dynamicType = <unset>,dynamicProperty = (vmodl.DynamicProperty) [],msg = 'vim.fault.InvalidState',faultCause = <unset>,faultMessage = (vmodl.LocalizableMessage) [],reason = 'Invalid fault'}2020-02-12T05:31:10.24Z INFO root Exiting with exit-code 1 In source hostd.log, you see message similar to: 2020-02-12T04:19:51.505Z info hostd[13947533] [Originator@6876 sub=Vimsvc.TaskManager opID=fb50909a user=root] Task Created : haTask--vim.SearchIndex.findByIp-54348 2020-02-12T04:19:51.514Z info hostd[13947442] [Originator@6876 sub=Default opID=fb50909a user=root] AdapterServer caught exception: N3Vim5Fault12InvalidState9ExceptionE(Fault cause: vim.fault.InvalidState--> ) --> [context]zKq7AVICAgAAAJA0nQATaG9zdGQAAAy3NWxpYnZtYWNvcmUuc28AAJDIGwDwuBcBg/ViaG9zdGQAAUQhagHSIWoB0hK1AQfBZAFg2GQBVeVkARbpZIJ2LAEBbGlidmltLXR5cGVzLnNvAAE1C8IBLRTCALVNKACTUSgAyyw2Azt9AGxpYnB0aHJlYWQuc28uMAAEHZsObGliYy5zby42AA==[/context]2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Vimsvc.TaskManager opID=fb50909a user=root] Task Completed : haTask--vim.SearchIndex.findByIp-54348 Status error 2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Activation [N5Vmomi10ActivationE:0x000000836d711860] : Invoke done [findByIp] on [vim.SearchIndex:ha-searchindex]2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg datacenter: --> (null) 2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg ip: --> "10.x.x.x" 2020-02-12T04:19:51.518Z verbose hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Arg vmSearch:--> true 2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Throw vim.fault.InvalidState2020-02-12T04:19:51.518Z info hostd[13947442] [Originator@6876 sub=Solo.Vmomi opID=fb50909a user=root] Result: -->(vim.fault.InvalidState) { -->faultCause=(vmodl.MethodFault) null, --> faultMessage = <unset> --> msg = "" --> } Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
During upgrade prechecks, a call to the SearchIndex.findByIp VIM API might encounter an invalid VM during iteration. At this point, an error will occur and the call will fail before checking all VMs (usually this occurs when the invalid VM is traversed before the VM with the desired IP). This will fail upgrade prechecks.
This is a known issue.Currently, there is no resolution.
To workaround this issue:Make sure there are no VMs on the host which would cause the call to fail. Specifically: Make sure there are no VMs that are in the process of being created/deleted from disk/added to the inventory/reloaded - such VMs may also raise the same error so wait until these operations are completed.Determine if there are any invalid VMs by running the command: vim-cmd vmsvc/getallvmsExample: a 'Skipping invalid VM' warning will be listed for all invalid VMs Skipping invalid VM '241'Vmid Name File Guest OS Version Examine the invalid VMs : VM's may be invalid due to issues with configuration, storage inaccessibility, missing disks/VMX files etc. If the issue cannot be remediated, make sure they do not interfere with the upgrade, unregister them from the host. For example, using vim-cmd: vim-cmd vmsvc/unregister 241Execute the unregister command for each invalid VM. Re-try the upgrade.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
