OPERATIONAL DEFECT DATABASE
...
...
ESXi hosts goes not responding in vCenter and recovers automatically few minutes later. In var/log/hostd.log you see the hostd process crashing with panic: 2019-10-29T03:35:29.797Z error hostd[309C2B70] [Originator@6876 sub=UW Memory checker] Current value 1288636 exceeds hard limit 1286881. Shutting down process.2019-10-29T03:35:29.797Z panic hostd[309C2B70] [Originator@6876 sub=Default]-->--> Panic: Memory exceeds hard limit. Panic--> Backtrace:--> [backtrace begin] product: VMware ESX, version: 6.5.0, build: build-6765664, tag: hostd, cpu: x86, os: esx, buildType: release---> In var/log/hostd.log you see the below messages referring to a corrupt VMware Tools windows.iso image Line 16733: 2019-10-29T04:02:20.035Z info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Selected Tools ISO 'windows.iso' for 'windows7srv-64' guest. Line 16744: 2019-10-29T04:02:20.059Z info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Error hash mismatch at offset 0x8000, expected hash [PbIrjdgFyu824nbRIbnFi8+DNkHIO06vZK/AIFcymIo=], but found [kBzqaLCzmb61r+AtU/ft/+kgRiR8V9OgDtOq6QQlQrI=] Line 16745: 2019-10-29T04:02:20.059Z info hostd[2C297B70] [Originator@6876 sub=Libs] ToolsISO: Invalid data at offset 8000 (bytes 2048) Line 16746: 2019-10-29T04:02:20.059Z info hostd[2C297B70] [Originator@6876 sub=Libs] Failed to open Tools ISO image.Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
To resolve the hostd issue.
The hash check fails for VMware Tools ISO and the hostd process fails to open/extract the ISO file. Due to this failure, the allocated hash info is not being cleaned up efficiently, leading to memory leak and the eventual hostd crash.
Requires host to be in maintenance mode.
VMware vSphere ESXi 6.5 P05, and VMware vSphere ESXi 6.7 P02 has the fix for the issue.
To workaround this issue follow the below steps.Copy the /productLocker/vmtools/ folder and its contents from a working host running the same ESXi version/build and replace them on the affected host.1. Enter the ESXi host in Maintenance Mode2. Enable SSH on both the "good" (for e.g. esxi01 ) and the "affected" (esxi02) host.3. Login as "root" user4. scp the /productLocker/vmtools/ folder and contents from the "good" host to the "problem" host$ [esxi01:~]$ scp -rf /productLocker/vmtools/ esxi02:/productLocker/Note: This will overwrite the vmtools folder on the problem host, hence you could backup this folder locally if you want but it is not mandatory for the purpose of this activity.5. Verify the MD5SUM of the vmtools folder contents on the good and bad host. These should match after copying.For e.g. md5sum /productLocker/vmtools/windows.iso1f13ee2eb9dfb4eab070c241a1d71cf4 /productLocker/vmtools/windows.isoNote: Run the md5sum command on all the files under the vmtools folder on both hosts5. Restart Management Agents services.sh restart6. Exit Maintenance Mode
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
