OPERATIONAL DEFECT DATABASE
...
...
ESXi hosts can go to non-responsive state in vCenterIn some occasions, they still are responsive, but some memory errors are seeing when executing esxcli commands, eg esxcli esxcli command list CRITICAL:root:Exception: CRITICAL:root:Traceback (most recent call last): File "/bin/esxcli", line 46, in <module> from pyVmomi import vmodl, vim, SoapAdapter, VmomiSupport, Cache, MemoryError In vmkernel.log have admission failure in path: ssh/phyton messages 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14635: Admission failure in path: ssh/python.4918545/uw.4918545 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14642: uw.4918545 (33688803) extraMin/extraFromParent: 64/64, ssh (748) childEmin/eMinLimit: 204785/204800 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14635: Admission failure in path: ssh/python.4918545/uw.4918545 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14642: uw.4918545 (33688803) extraMin/extraFromParent: 64/64, ssh (748) childEmin/eMinLimit: 204785/204800 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14635: Admission failure in path: ssh/python.4918545/uw.4918545 2020-07-13T18:47:16.296Z cpu2:4918545)MemSched: 14642: uw.4918545 (33688803) extraMin/extraFromParent: 64/64, ssh (748) childEmin/eMinLimit: 204785/204800 In vmkwarning.log we can see messages similar to the following 2020-07-08T20:55:41.975Z cpu2:2757371)WARNING: MemSched: 11696: Group vsanperfsvc: Requested memory limit 0 KB insufficient to support effective reservation 7500 KB 2020-07-08T21:21:52.171Z cpu10:2760381)WARNING: UserSocketInet: 2244: python: waiters list not empty! 2020-07-08T23:37:16.500Z cpu21:2790677)WARNING: LinuxThread: 381: python: Error cloning thread: -28 (bad0081) 2020-07-08T23:39:58.661Z cpu39:2791235)WARNING: UserParam: 1326: busybox: could not change group to <host/vim/vimuser/terminal/ssh>: Admission check failed for memory resource 2020-07-08T23:39:58.661Z cpu39:2791235)WARNING: LinuxFileDesc: 6270: busybox: Unrecoverable exec failure: Failure during exec while original state already lost CPU usage in the host is very high. e.g.Running uptime shows the load average very high [root@esxi01:~]uptime12:11:08 up 14 days, 21:28:56, load average: 0.94, 0.95, 0.95 Note:The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Avoid ESXi host to go into not responding status.
By default, the hostd service retains completed tasks for 10 minutes.If too many tasks come at the same time, for instance calls to get the current system time from the ServiceInstance managed object, hostd might not be able to process them all and fail with an out of memory message.
None
This issue is resolved in VMware vSphere 6.5 Patch ESXi650-202007001.To download go to the Customer Connect Patch Downloads page.This issue is resolved in VMware vSphere 6.7 Patch ESXi670-202008001.To download go to the Customer Connect Patch Downloads page.
To workaround this issue Make fewer such vim.ServiceInstance.* calls in quick succession.Lower the value of the taskRetentionInMins option in hostd's /etc/vmware/hostd/config.xml (default is 10 minutes) by following below steps /etc/init.d/hostd stopedit /etc/vmware/hostd/config.xmlBefore: <!-- <taskRetentionInMins> 10 </taskRetentionInMins> -->After: <taskRetentionInMins> 5 </taskRetentionInMins>/etc/init.d/hostd start
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
