Symptoms
ESXi Host is reporting an alarm
Alarm 'Host error' on <Host Name> triggered by event 30258576 'Issue detected on <Host Name> in <Cluster Name>: vmsyslog logger <ip>:514 lost 2210935 log messages
In /var/log/vmkernel.log similar entries are seen
2020-04-09T17:25:34.490Z cpu24:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4810784/uw.48107842020-04-09T17:28:14.367Z cpu29:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4810998/uw.48109982020-04-09T17:41:33.482Z cpu0:65990)MemSched: 14635: Admission failure in path: logging/vmsyslogd.4811898/uw.48118982020-04-08T13:33:12.334Z cpu0:65990)ALERT: vmsyslog logger <IP address>:514 lost 2210935 log messages
In /var/log/.vmsyslogd.err similar entries are seen
Traceback (most recent call last): File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/site-packages/vmsyslog/loggers/file.py", line 379, in writeLog File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 676, in __init__ File "/build/mts/release/bora-15256549/bora/build/esx/release/vmvisor/sys-boot/lib64/python3.5/subprocess.py", line 1228, in _execute_childOSError: [Errno 28] No space left on deviceNote: The preceding log excerpts are only examples.Date,time and environmental variables may vary depending on your environment.
Purpose
To stop the out of memory condition for logging group.
Cause
vmsyslog is spawning subprocess frequently to execute this on-rotate command. This is causing memory exhaustion, resulting in memory limit reaching the max configured for ‘logging’ group.
Resolution
This issue is resolved in
VMware vSphere ESXi 6.7 Patch release ESXi670-202004002VMware vSphere ESXi 6.5 Patch release ESXi650-202007001
These patches are available to download at Customer Connect Patch Downloads
Workaround
To workaround this issue1. Verify the memory limits:# localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group getmemconfig -g host/vim/vmvisor/loggingExamplerMinPeak Is greater than eMinPeak for logging:
gid name parGid eMinPeak rMinPeak
752 logging 14 87040 94952
14 vmvisor 2 2999688 2999688
2.Increase the memory allocation for the pool logging to maximum 120mb using following command.localcli --plugin-dir=/usr/lib/vmware/esxcli/int sched group setmemconfig -g host/vim/vmvisor/logging -i 82 -m 120 -u mbNote: The workaround is not persistent across reboot since /usr/lib/vmware/rp/bin/configRP script is called after reboot which will restore default configs as default values are hardcoded which cannot be modified.
