Loading...
Loading...
From the SDDC Manager - ESXi upgrades fail on stage: ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP.The task does not show up in the vCenter task and events.We see the following entries for the LCM logs on the SDDC Manager in /var/log/vmware/vcf/lcmlcm-debug.log: 022-03-01T14:51:35.897+0000 INFO [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-4cf6-a489-403f690f87a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-44b7-9ced-51bb02ca84f4] [c.v.e.s.l.p.i.e.EsxVumUpdateStageRunnerImpl,pool-3-th read-22] The stage: ESX_UPGRADE_VUM_STAGE_ATTACH_BASELINE_GROUP returned status: ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH, status msg: Exception when assigning baseline group to cluster: integrity.fault.NoVcConnection.summary 2022-03-01T14:51:35.897+0000 INFO [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-4cf6-a489-403f690f87a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-44b7-9ced-51bb02ca84f4] [c.v.e.s.l.p.i.e.EsxVumUpdateStageRunnerImpl,pool-3-th read-22] esxHostUpgradeStatus :ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH 2022-03-01T14:51:35.897+0000 ERROR [vcf_lcm,0000000000000000,0000,precheckId=282f4d4a-d401-4cf6-a489-403f690f87a6,resourceType=BUNDLE,resourceId=aa7b16b1-d719-44b7-9ced-51bb02ca84f4] [c.v.e.s.l.p.i.e.EsxVumPrimitiveImpl,pool-3-thread-22] Error occurred performing the bundle precheck for cluster: e1f7c485-281c-4027-82c2-0dcdffc82761 and bundle: aa7b16b1-d719-44b7-9ced-51bb02ca84f4 java.lang.IllegalArgumentException: No enum constant com.vmware.evo.sddc.lcm.primitive.model.esx.PrecheckErrorCode.ESX_UPGRADE_VUM_FAILED_BASELINE_GROUP_ATTACH at java.lang.Enum.valueOf(Enum.java:238) at com.vmware.evo.sddc.lcm.primitive.model.esx.PrecheckErrorCode.valueOf(PrecheckErrorCode.java:8) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxUtils.getPrecheckError(EsxUtils.java:1428) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumUpdateStageRunnerImpl.precheckUploadAndScan(EsxVumUpdateStageRunnerImpl.java:755) at com.vmware.evo.sddc.lcm.primitive.impl.esx.EsxVumUpdateStageRunnerImpl.doBundlePrecheck(EsxVumUpdateStageRunnerImpl.java:666) ... ... 2022-03-01T14:41:10.976+0000 INFO [vcf_lcm,651a8746a99c92f1,d931] [c.v.e.sddc.lcm.audit.AuditScheduler,pool-6-thread-8] Last upgrade status for software type ESX_HOST and resourceId 173796b2-1f4d-4e1f-afa8-a07733f8cb9f is: {"id":"91","bundleId":"aa7b16b1-d719-44b7-9ced-51bb02ca84f4","bundleElementId":"583db34c-2981-4cea-bb58-f47eaae6f8c7","bundleElementOrder":1,"resourceType":"ESX_HOST","resourceId":"173796b2-1f4d-4e1f-afa8-a07733f8cb9f","upgradeId":"30084d8d-e9ee-4216-984c-fa809c4f9c8d","upgradeStatus":"COMPLETED_WITH_FAILURE","upgradeError":{"errorType":"RECOVERABLE","stage":"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP","errorCode":"com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception","errorDescription":"Failed to create baseline group: Database temporarily unavailable or has network problems.","errorDescriptionCodes":["com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception"],"errorDescriptionAttributes":{"0":"Database temporarily unavailable or has network problems."},"metadata":"\nManual intervention needed as upgrade failed during install stage.\nCheck for errors in the lcm log files located on SDDC Manager under /var/log/vmware/vcf/lcm.\n Please retry the upgrade once the upgrade is available again.","metadataCodes":["com.vmware.vcf.error.runtime.esx.update.vum.baselinegroup.creation.exception.remedy","MANUAL_INTERVENTION","LCM_LOG","RETRY_NOTE"],"metadataAttributes":{"LCM_LOG_LOCATION":"/var/log/vmware/vcf/lcm","LCM_HOST_ADDRESS":"127.0.0.1","BACKUP_REPO_USERNAME":"backup","BACKUP_REPO_SSH_ADDRESS":"localhost","UPGRADE_TARGET_USERNAME":"svc-vcf-wh-0000002941","UPGRADE_TARGET_SSH_ADDRESS":"esx.mycompany.net"}},"startTime":1646145500980,"endTime":1646145532882,"numRetriesLeft":2,"upgradeStage":"[\n \"ESX_UPGRADE_VUM_STAGE_PRECHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_CUSTOM_IMAGE_SPEC\",\n \"ESX_UPGRADE_VUM_STAGE_UPLOAD_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\"\n]","backupInfoList":[],"upgradeStages":"[\n \"ESX_UPGRADE_VUM_STAGE_PRECHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_CUSTOM_IMAGE_SPEC\",\n \"ESX_UPGRADE_VUM_STAGE_ADD_HOST_TO_VC_INVENTORY\",\n \"ESX_UPGRADE_VUM_STAGE_UPLOAD_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_REGISTER_UPLOADED_FILES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINES\",\n \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\",\n \"ESX_UPGRADE_VUM_STAGE_ATTACH_BASELINE_GROUP\",\n \"ESX_UPGRADE_VUM_STAGE_SCAN_HOST\",\n \"ESX_UPGRADE_VUM_STAGE_TAKE_BACKUP\",\n \"ESX_UPGRADE_VUM_STAGE_DISABLE_DRS_RULES\",\n \"ESX_UPGRADE_VUM_STAGE_INSTALL_UPDATE\",\n \"ESX_UPGRADE_VUM_STAGE_ENABLE_DRS_RULES\",\n \"ESX_UPGRADE_VUM_STAGE_CLEANUP\",\n \"ESX_UPGRADE_VUM_STAGE_REMOVE_HOST_FROM_VC_INVENTORY\",\n \"ESX_UPGRADE_VUM_STAGE_POSTCHECK\",\n \"ESX_UPGRADE_VUM_STAGE_SUCCESS\"\n]","metadata":"{\n \"stage\": \"ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP\",\n \"primitiveSpec\": {\n \"upgradeId\": \"30084d8d-e9ee-4216-984c-fa809c4f9c8d\",\n \"resourceId\": \"173796b2-1f4d-4e1f-afa8-a07733f8cb9f\",\n \"resourceType\": \"ESX_HOST\",\n \"bundleElementOrder\": 1,\n \"patchElement\": {\n \"id\": \"583db34c-2981-4cea-bb58-f47eaae6f8c7\",\n \"type\": \"ESX_HOST\",\n \"vendor\": \"VMware\",\n \"version\": \"7.0.1-18150133\",\n \"previousVersion\": \"7.0.1-17551050\",\n \"file\": \"aa7b16b1-d719-44b7-9ced-51bb02ca84f4/bundle-44636/VMware-VMvisor-Installer-7.0hp-18150133.x86_64.iso\",\n \"checksum\": \"11ac4d99500c56fb9cdc20d912a777b52fd4fbc0daa21a6f30bd42a4e482f3c5\"\n },\n \"patchElements\": [],\n \"slaType\": \"SLOW\",\n \"endTime\": 1646159900972\n },\n \"esxVumUpdateStateObjects\": {\n \"upgradeId\": \"30084d8d-e9ee-4216-984c-fa809c4f9c8d\",\n \"updateFiles\": [\n \"/nfs/vmware/vcf/nfs-mount/vendor-isos/RKN-LVO-ESXI-7.0.1_18150133-LVO.701.10.3.iso\"\n ],\n \"remoteFilePathToLcmLocalFileCheckSumMap\": {},\n \"desiredVersion\": \"7.0.1-18150133\",\n \"upgradeBaselineId\": -1,\n \"patchBaselineIds\": [],\n \"baselineGroupId\": -1,\n \"esxMOID\": \"host-21\",\n \"patchKeys\": [],\n \"upgradeKey\": 2,\n \"upgradeIsoCheckSum\": \"d9094441c262e30060308d437aa0cb223eb5bbac8eaf6d081539a95bd016eff4\",\n \"drsRules\": [],\n \"isStandaloneHost\": false,\n \"shutdownVMs\": false,\n \"totalDisksCount\": 0,\n \"esxCustomImageSpec\": {\n \"bundleId\": \"aa7b16b1-d719-44b7-9ced-51bb02ca84f4\",\n \"targetEsxVersion\": \"7.0.1-18150133\",\n \"useVcfBundle\": false,\n \"customIsoAbsolutePath\": \"/nfs/vmware/vcf/nfs-mount/vendor-isos/RKN-LVO-ESXI-7.0.1_18150133-LVO.701.10.3.iso\"\n },\n \"vumBundleUploadLockHeld\": true\n }\n}","stages":[]} VMware Update Manager logs on the vCenter Server report the following in the /var/log/vmware/vmware-updatemgr/vum-server/vmware-vum-server.log: [VdbStatement] SQLError was thrown: "ODBC error: (23503) - ERROR: insert or update on table "vci_baseline_group_baselines" violates foreign key constraint "fk_vci_blgp_bl_ref_bl"; [#132] --> Error while executing the query" is returned when executing SQL statement "INSERT INTO VCI_BASELINE_GROUP_BASELINES (baseline_group_id, baseline_group_version, baseline_id) (SELECT ? AS baseline_group_id, ? AS baseline_group_ version, ? AS baseline_id FROM VCI_BASELINE_GROUP_BASELINES WHERE baseline_group_id = ? AND baseline_group_version = ? AND baseline_id = ? HAVING COUNT(*) = 0)" [#133] 2022-03-01T15:20:35.062Z info vmware-vum-server[59216] [Originator@6876 sub=VcIntegrity] Session: [sessionId masked]; user VSPHERE.LOCAL\Administrator [#133] 2022-03-01T15:20:35.064Z info vmware-vum-server[59216] [Originator@6876 sub=VcIntegrity] Active? true 2022-03-01T15:20:35.070Z info vmware-vum-server[58938] [Originator@6876 sub=VcIntegrity] Connecting to host <esxi_host_fqdn> on port 80 using protocol http
The purpose of this document is to workaround the ESXi upgrade failing with error: ESX_UPGRADE_VUM_STAGE_CREATE_BASELINE_GROUP, and successfully complete the upgrades.
This issue is caused due to the fact that the LCM Baseline is not detached or cleaned up after a host scan during precheck or after a remediation task.
MODERATE: The workaround involves resetting the VUM DB on the vCenter Server, as well as truncating a table on the SDDC DB. Offline snapshots of all VCs in the SSO, as well as a snapshot of the SDDC Manager are highly recommended.
This issue is resolved in VMware Cloud Foundation 4.3.
1.Take a snapshot of both the SDDC Manager and the vCenter.2. Reset the VUM DB on the vCenter using the steps provided in KB: https://kb.vmware.com/s/article/21472843. Login to SDDC Manager via SSH using the vcf user and then su root.4. Run following command to truncate the "vum_bundle_upload" table on the SDDC: psql -U postgres -h localhost lcm -c "truncate vum_bundle_upload" 5. Retry upgrade from the SDDC Manager UI.
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.