Symptoms
Cannot power on a virtual machine with 850 GB plus memory + PCI Passthrough devicevirtual machine power on failed with 1 TB memory and PCI passthrough device "vmkernel: Out of memory"In the var/log/vmkernel.log, you see entries similar to:
cpu9:327411)WARNING: Heap: 3672: Heap mainHeap (30819192/31462264): Maximum allowed growth (643072) too small for size (4198400)cpu9:327411)WARNING: Heap: 4169: Heap_Align(mainHeap, 4194312/4194312 bytes, 8 align) failed. caller: 0x41801af5a178cpu9:327411)VSCSI: 6541: handle 8207(vscsi0:0):Destroying Device for world 327412 (pendCom 0)cpu46:68106)Config: 706: "SIOControlFlag2" = 0, Old Value: 1, (Status: 0x0)cpu42:66360)Tcpip_Flow: 129: Failed to pin vmk0 rxDisp 0 to world 328437 (awk): Not initialized
In the vmware.log, you see entries similar to:
vmx| I125+ Power on failure messages: Passthrough device 'pciPassthru0' vGPU 'grid_p40-4q' disallowed by vmkernel: Out of memory. vmx| I125+ Could not initialize plugin '/usr/lib64/vmware/plugin/libnvidia-vgx.so' for vGPU 'grid_p40-4q'. vmx| I125+ Module 'DevicePowerOn' power on failed. vmx| I125+ Failed to start the virtual machine.
Cause
This issue occurs when vmkernel is unable to allocate memory from the 'mainHeap' during PCI passthrough device registration. It fails due to not enough memory in mainHeap.
Impact / Risks
Cannot power on VM with 1 TB Memory when PCI passthrough device is attached.
Resolution
This issue is resolved in:
ESXi600-201807001 available at VMware Downloads.ESXi 6.5 Update 2, available at VMware Downloads.
Workaround
To workaround the issue if you do not want to upgrade, decrease the virtual machine memory below 850 GB.
