OPERATIONAL DEFECT DATABASE
...
...
HA cannot be successfully enabled on the newly added/moved host in any one of the following cases: A host with ESXi 7.0u3/u3a is added/moved to a HA-enabled VUM clusterA host with ESXi 7.0u3/u3a is added/moved to a non-HA-enabled VUM cluster and HA is enabled later on the cluster.A host with ESXi 7.0u2c/u2d or ESXi 7.0u3/u3a is added/moved to a HA-enabled VLCM clusterA host with ESXi 7.0u2c/u2d or ESXi 7.0u3/u3a is added/moved to a non-HA-enabled VLCM cluster and HA is enabled later on the cluster.Multiple hosts (at least one 7.0u3c host and at least one host with ESXi 7.0u2c/u2d or ESXi 7.0u3/u3a) are added to a HA-enabled VLCM cluster and the host maintenance mode is exited.A previously disconnected host with ESXi 7.0u3/u3a is reconnected in a HA-enabled VUM cluster.A previously disconnected host with ESXi 7.0u2c/u2d or ESXi 7.0u3/u3a is reconnected in a HA-enabled VLCM cluster When trying to enable HA, the installation of the HA agent (FDM) on the host fails.In some cases we also observe a symptom where in "Remediate HA" task will complete but HA is not enabled on host. HA is not healthy on hostThe HA status for the ESXi host is stuck at "Configuration Error" with the description "An error occurred when vCenter Server attempted to initialize the vSphere HA Agent running on the host."Sometimes, the HA status for the ESXi host is stuck at "HA Agent Unreachable" with the description "The vSphere HA Agent on the host cannot be reached."In some cases the "Configuring vSphere HA" task fails with "Cannot complete the configuration of the vSphere HA agent on the host. "Applying HA VIBs on the cluster encountered failure"".In some cases the "Remediate HA" task fails with "A general system error occurred: Installing HA components failed on the cluster: domain-c100".Migrating or powering on VMs on this newly added host in the HA enabled cluster fail.VLCM cluster Image Apply on the HA-enabled VLCM cluster will fail because HA is unhealthy.When the host is being moved/added to a VLCM cluster, the following entries will be seen in /var/run/log/lifecycle.log on the ESXi. lifecycle.log 2022-01-10T15:55:07Z lifecycle: 308060: ImageProfile:844 INFO Adding VIB VMware_bootbank_vmware-fdm_7.0.3-19012297 to ImageProfile (Updated) ESXi-7.0U2d-18538813-standard 2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:147 ERROR [ValueError] Expected 1 component, found 22022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR Traceback (most recent call last):2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR File "/usr/lib/vmware/lifecycle/bin/imagemanagerctl.py", line 665, in components2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR Transaction(initInstallers=False).InstallComponentsFromSources(2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 532, in InstallComponentsFromSources2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR _checkComponentDowngrades(curProfile, newProfile,2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 2264, in _checkComponentDowngrades2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR compDowngrades = curProfile.GetCompsDowngradeInfo(newProfile)2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2376, in GetCompsDowngradeInfo2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR curComp = self.components.GetComponent(name)2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR File "/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1243, in GetComponent2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR raise ValueError('Expected 1 component, found %u'2022-01-10T15:55:07Z lifecycle: 308060: imagemanagerctl:152 ERROR ValueError: Expected 1 component, found 2 When the host is being moved/added to a VUM cluster, the following entries will be seen in /var/run/log/esxupdate.log on the ESXi. esxupdate.log 2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: An unexpected exception was caught:2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: Traceback (most recent call last):2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/usr/sbin/esxupdate", line 216, in main2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: cmd.Run()2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esx5update/Cmdline.py", line 153, in Run2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 965, in InstallVibsFromSources2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: inst, removed, exitstate = self._installVibs(curprofile,2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1207, in _installVibs2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: hasConfigDowngrade = checkFdmConfigDowngrade(curProfile, newProfile)2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Transaction.py", line 1122, in checkFdmConfigDowngrade2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: compDowngrades = curProfile.GetCompsDowngradeInfo(newProfile)2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/ImageProfile.py", line 2416, in GetCompsDowngradeInfo2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: curComp = self.components.GetComponent(name)2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: File "/lib64/python3.8/site-packages/vmware/esximage/Bulletin.py", line 1276, in GetComponent2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: raise ValueError('Expected 1 component, found %u'2022-01-10T17:37:04Z esxupdate: 136643: esxupdate: ERROR: ValueError: Expected 1 component, found 2 Environment has HA enabled VUM cluster This cluster contains vSphere 7.0 u2c/u2d/u3/u3a host which has 2-component (i40en & i40enu) situationThe user transitions this cluster to vLCMPost transition HA will invoke Apply HA API which will fail. Because of this failure HA will raise alarms but the vSphere HA will be still functional
This issue occurs when both the i40en and i40enu VIBs for the inbox Intel network driver are installed on the host. In vSphere 7.0 Update 2, the driver was renamed from i40en to i40enu. Starting with vSphere 7.0 Update 3, the inbox i40enu network driver for ESXi changes back to i40en. The HA VIB Installation fails on the affected host.
This issue is resolved in 7.0 U3C
For VLCM cluster, use any one of the following workarounds on the hosts that have the issue: Remove host from vLCM cluster, and upgrade to ESXi 7.0u3c via VUM, then add/move the host back to the vLCM cluster.If you do not want to upgrade the hosts, do a manual cleanup. Disable vSphere HA on the cluster with the affected hosts.On each host that has the issue: Put the ESXi in maintenance mode.Run the following command to remove the obsolete i40enu VIB:esxcli software vib remove --vibname=i40enuOnce the VIB has been removed, reboot the ESXi host. When this has been done for all hosts in the cluster which have the issue, HA can be enabled again on the cluster. Disable vSphere HA on the cluster, have a valid desired image of ESXi 7.0u3c, and continue with Image Apply to upgrade the hosts which have the issue to ESXi 7.0u3c. This should turn HA status back to healthy. Note: For VUM cluster, remediate the hosts which have the issue via rollup upgrade or ISO baseline upgrade to ESXi 7.0u3c.NOTE :- The below applies to the issue reported to Point number#12 under the Symptoms section Do not proceed with a vCenter upgrade (to any vCenter release) otherwise that will cause HA to be completely non-functionalUpgrade the ESXi to vSphere 7.0u3cFirst update the vSphere 7.0 U2c/ U2d/ U3/ U3a host to 7.0u3c via vLCMOnly post completion of step#2 mentioned above the Environment can proceed with vCenter upgrade
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
