Symptoms
You experience these symptoms on an ESXi 7.0 U1 or 7.0U2 vSAN enabled host
Physical disk health retrieval issues with error Host XXX - Error: Out of MemoryAdvanced vSAN Configuration in sync - vSAN daemon liveness - CLOMD status: AbnormalInaccurate health check results due to vsanmgmtd memory leak
In the /var/run/log/vmkernel.log files on the ESXi host, you see messages similar to:
2021-01-20T09:33:12.993Z cpu40:13487007)Admission failure in path: host/vim/vmvisor/vsanperfsvc:cmmds-tool.13487007:uw.134870072021-01-20T09:33:12.993Z cpu40:13487007)Admission failure in path: host/vim/vmvisor/vsanperfsvc:cmmds-tool.13487007:uw.13487007
In the var/run/log/vsanmgmt.log file, you see messages similar to:
2021-01-20T03:33:55.754Z info vsand[2510678] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=171.324MB, eMinPeak=173.968MB, rMinPeak=174.984MB MEMORY PRESSURE2021-01-20T04:03:59.304Z info vsand[2510678] [opID=MainThread statsdaemon::_logDaemonMemoryStats] Daemon memory stats: eMin=170.428MB, eMinPeak=173.968MB, rMinPeak=174.984MB MEMORY PRESSURE
In the /var/run/log/vmsyslogd-dropped.log file, you see messages similar to:
2020-09-25T04:38:38.257Z: 2020-09-25T04:38:38.021Z cpu51:18265550 opID=34b30f7d)Admission failure in path: host/vim/vmvisor/vsanperfsvc:vsanmgmtd.18265531:uw.18265531
Note: The timestamp and memory usage will vary for different clusters.The vsanmgmtd core dump file (vsanmgmtd-zdump.XXX) can be found under /var/core/
Purpose
Identify and resolve issues with ESXi host(s) hitting a vsanmgmt memory leak.
Cause
Memory leak issue in vsanmgmtd service affecting ESXi 7.0 U1 and 7.0U2. This issue only affects the vSAN management stack.
Impact / Risks
If the vsanmgmtd encounters a memory leak it may cause inaccurate health check results which are only resolved by restarting vsanmgmt or rebooting the ESXi Host.
Resolution
This issue is resolved by upgrading both vCenter and ESXi to 7.0U3c or later.
Workaround
If you are unable to upgrade, you can work around the issue by restarting the vsanmgmt service on the affected ESXi host(s) via SSH: # /etc/init.d/vsanmgmtd restart
