Loading...
Loading...
ESXi hosts keep disconnecting and marked as not responding following the upgrade of vCenter Server to 8.0U2. After restarting the vpxa and hostd services on the ESXI host, it will reconnect for some, time then come back as disconnected. ESXi host version is earlier than 8.0U2 Vpxd.log on the vCenter Server: /var/log/vmware/vpxd/vpxd.log 2023-09-24T05:04:35.424-04:00 info vpxd[06067] [Originator@6876 sub=certmgrLogger opID=HB-host-12345@555-1234567a-WorkQueue-373fc90c] Will update root certificates on host ; [vim.HostSystem:host-12345,esxi.hostname.local], on vc: (string) [ 2023-09-24T05:04:35.457-04:00 info vpxd[07207] [Originator@6876 sub=vpxLro opID=HB-host-12345@555-1234567a-04] [VpxLRO] -- BEGIN lro-119514 -- -- AddClusterStoreMember -- 2023-09-24T05:04:35.586-04:00 warning vpxd[06103] [Originator@6876 sub=vmomi.soapStub[1107] opID=HB-host-12345@555-1234567a-DvsHandleHostReconnect-4b604375] SOAP request returned HTTP failure; <<io_obj p:0x00007fcc2c34f628, h:130, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-12345/vpxa>, method: commitTransaction; code: 500(Internal Server Error) 2023-09-24T05:04:35.586-04:00 error vpxd[06103] [Originator@6876 sub=hostMethod opID=HB-host-12345@555-1234567a-DvsHandleHostReconnect-4b604375] Commit call for method [applyDvs] transaction Id [159] failed on host [[vim.HostSystem:host-12345,esxi.hostname.local]] with exception:[(vmodl.fault.HostCommunication) { --> faultCause = (vmodl.MethodFault) null, --> faultMessage = <unset> --> msg = "Received SOAP response fault from [<<io_obj p:0x00007fcc2c34f628, h:130, <UNIX ''>, <UNIX '/var/run/envoy-hgw/hgw-pipe'>>, /hgw/host-12345/vpxa>]: commitTransaction --> " --> }] Vpxa.log on the ESXi host: /var/run/log/vpxa/vpxa.log 2023-09-24T09:04:35.584Z Er(163) Vpxa[6122473]: [Originator@6876 sub=Default opID=37f668e7] [VpxLRO] -- ERROR lro-80 -- 52839d49-b2ee-8cf2-70fc-3dfdc4ce24da -- networkSystem -- vim.host.NetworkSystem.commitTransaction: :vmodl.fault.HostCommunication 2023-09-24T09:04:35.584Z Er(163) Vpxa[6122370]: --> Result: 2023-09-24T09:04:35.584Z Er(163) Vpxa[6122370]: --> (vmodl.fault.HostCommunication) { 2023-09-24T09:07:14.550Z In(166) Vpxa[6128218]: [Originator@6876 sub=vpxaInvtHost opID=WFU-3252723e] ServerId has been changed from 805152 to 0 2023-09-24T09:07:14.550Z Er(163) Vpxa[6128218]: [Originator@6876 sub=vpxaInvtHostCnx opID=WFU-3252723e] Can't connect to hostd. Shutting down... 2023-09-24T09:07:14.550Z In(166) Vpxa[6128218]: [Originator@6876 sub=Default opID=WFU-3252723e] [Vpxa] Shutting down now Hostd.log on the ESXi host: /var/run/log/hostd.log 2023-09-24T09:04:35.435Z In(166) Hostd[6040281]: [Originator@6876 sub=Libs opID=HB-host-12345@555-1234567a-02-18-df27 sid=52b461fa user=vpxuser:<no user>] info [ConfigStore:ee32fc6700] [cs:4:1947917405]Transaction committed,level = 1 2023-09-24T09:04:35.435Z In(166) Hostd[6040278]: [Originator@6876 sub=Vimsvc.CertMgr opID=HB-host-12345@555-1234567a-WorkQueue-373fc90c-df29 sid=5216f1ca user=vpxuser] Discarding non-CA certificate: -----BEGIN CERTIFICATE-----
vCenter Server pushes certificate updates to the ESXi host on reconnect after the upgrade. If there is a non-CA certificate in the TRUSTED_ROOTS certificate store, the hostd discards this certificate and issues an ssl_reset . This causes vpxa to restart on ESXi hosts prior to 8.0U2. After the vpxa restart, vCenter Server again pushes the certificate updates to the ESXi host. Reconnecting causes the same behavior to repeat and the host to disconnect from vCenter.
Workaround 1 – Remove non‑CA certificates from the TRUSTED_ROOTS store (preferred) Step‑by‑step procedure Log in to the VCSA via SSH as the root user. List all entries in the TRUSTED_ROOTS store and display their key usage: # /usr/lib/vmware-vmafd/bin/vecs-cli entry list --store TRUSTED_ROOTS --text | egrep 'Alias|Key Usage' -A 1 | egrep -v 'Entry type|--' To remove any non-CA certificates, follow the steps outlined in Broadcom article: Removing Expired CA Certificates from the TRUSTED_ROOTS store in the VMware Endpoint Certificate Store(VECS) Workaround 2 – Allow any certificate on the ESXi host (use only if the certificate must be retained) This approach modifies an advanced setting on each affected ESXi host to permit the host to accept the non‑CA certificate without triggering an SSL reset. Step‑by‑step procedure Open the vSphere Client and connect directly to the ESXi host using its IP address or FQDN. Navigate to Manage → System → Advanced Settings . In the filter box, type Config.HostAgent.ssl.keyStore.allowAny Set the value of Config.HostAgent.ssl.keyStore.allowAny to true Click OK to save the change. Restart the hostd and vpxa services on the ESXi host to apply the new setting: # /etc/init.d/hostd restart # /etc/init.d/vpxa restart Caution: Enabling Config.HostAgent.ssl.keyStore.allowAny reduces the security posture of the host
Click on a version to see all relevant bugs
Dell Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.