Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
On vSphere 7.0 and 7.0b user may see message like “Heap vmfs3 already at its maximum size' In the vmkernel.log. Example In "vmkwarning.*" files user may see messages such as vmkwarning.0:2020-06-16T13:28:23.291Z cpu48:3479102)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.vmkwarning.0:2020-06-16T14:20:23.676Z cpu62:3479103)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.vmkwarning.0:2020-06-17T08:05:54.057Z cpu92:3909762)WARNING: Heap: 3651: Heap vmfs3 already at its maximum size. Cannot expand.Example In "vmkernel.*" files user may see messages such as 2020-06-29T14:59:36.351Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory2020-06-29T14:59:36.351Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory2020-06-29T14:59:36.356Z cpu21:5630454)WARNING: HBX: 2439: Failed to initialize VMFS distributed locking on volume 5eb9e8f1-f4aeef84-4256-1c34da50d370: Out of memory2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 1 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 0 gen 0 :Out of memory2020-06-29T14:59:36.356Z cpu21:5630454)Vol3: 4202: Failed to get object 28 type 2 uuid 5eb9e8f1-f4aeef84-4256-1c34da50d370 FD 4 gen 1 :Out of memory Datastores are showing "Not consumed" on hosts.VMs may fail to vMotion.VMs may become orphaned when powered off.Snapshot based backups may fail.In vCenter/hosts snapshot creation and consolidation may fail with: message = "Consolidation failed for disk node 'scsi0:1': 12 (Cannot allocate memory)."
Purpose of this article is to let the users know of a VMFS heap related issue present in vSphere 7.0 and 7.0b releases.
This is a known issue in VMFS6 where in certain work flows we allocate memory but do not free it up resulting in VMFS heap exhaustion. The memory allocated gets freed during VMFS resource allocation and hence creating a thick file on VMFS datastore is an effective work around. See work around section below for more details.
This issue is resolved in vsphere ESXi 7.0 Update 1.To workaround this issue follow the steps below: Create Eager zeroed thick disk on all of the mounted VMFS6 datastores. vmkfstools -c 10M -d eagerzeroedthick /vmfs/volumes/datastore/eztDisk Delete the Eager zeroed thick disk created in step 1. vmkfstools -U /vmfs/volumes/datastore/eztDiskWorkaround has to be done for each datastore on each host.If you need further support contact VMware Technical Support. For more information see How to file a Support Request in Customer Connect.
VMware Skyline Health Diagnostics for vSphere - FAQ