Purpose

Impact / Risks

Symptoms

After upgrading a host to ESXi 5.1 Update 2, virtual machines using the E1000/E1000E virtual adapter stop passing network traffic intermittently leading to the host being non-responsive After powering up a virtual machine, the network adapter remains disconnected While using vCloud Director, virtual machines spontaneously disconnect When reconnecting a disconnected virtual machine, the other powered on and running virtual machines on the host disconnect In the vmkernel.log file (located in /var/log) of the affected ESXi 5.1 U2 host, you see messages similar to:WARNING: Heap: 2677: Heap netGPHeap already at its maximum size. Cannot expand.WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800915066cWARNING: E1000: 8817: failed to enable port 0x200002f on DvsPortset-0: Out of memoryWARNING: Net: vm 367615: 4454: cannot enable port 0x200002f: Out of memory In the vmware.log file on the affected virtual machine, you see entries similar to:vcpu-0| I120: VMM initialized.vcpu-0| W110: Restoring VMMvcpu-0| I120: Msg_Post: Errorvcpu-0| I120: [msg.ethernet.e1000.openFailed] Failed to connect ethernet0.vcpu-0| I120: ----------------------------------------vcpu-0| I120: SVGA: Registering MemSpace at 0xd8000000(0xd8000000) and 0xd0800000(0xd0800000) In the hostd.log file (located in /var/log) of the affected ESXi 5.1 U2 host, you see entries similar to:[7F7C#### verbose 'Vmsvc.vm:/vmfs/volumes/4f4eb4eb-xxxxxxxx-xxxx-0017a4770020/VM/VM.vmx'] Handling message _vmx5: Failed to connect virtual device Ethernet0. In the hostd.log file, you see entries similar to:PktYYYY-MM-DDT<time>.905Z cpu24:12476)WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth PktYYYY-MM-DDT<time>.979Z cpu25:12478)WARNING: Heap: 3058: Heap_Align(netGPHeap, 35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>EthPkt2014-03-05T10:34:19.058Z cpu14:9175)WARNING: Heap: 3058: Heap_Align(netGPHeap,35156/35156 bytes, 8 align) failed. caller: 0x41800e15066c<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Eth Pkt<7>Et</time></time>Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.Note: The issue described in this article is specific to hosts running ESXi 5.1 Update 2. If you are running an older version of ESXi, your issue is not related to this article. Search for your symptoms in the Knowledge Base or file a Support Request. For more information, see Filing a Support Request in Customer Connect (2006985).Note: You may also get a similar error from a Nexus 1000v or VMware VDS even if you just list the vswitch(s) with the esxcfg-vswitch -l command. For more information, see Error message is displayed when a large number of dvPorts are in use in VMware ESXi 5.1.x (2034073).

Cause

This issue occurs when there is a memory leak in the E1000 VMkernel back end in ESXi 5.1 Update 2.

Resolution

This issue is resolved in VMware ESXi 5.1 Patch Release ESXi510-201402001 (Build 1612806). You can download the patch at http://www.vmware.com/patch/download/.For more information on the patch, see VMware ESXi 5.1, Patch Release ESXi510-201402001 (2072652).To work around this issue when you are unable to apply the patch, use one of these methods: Disable the TSO Zero Copy and TxZero Copy offloading features:Notes: This procedure does not reclaim memory lost to the leak. To recover lost memory, reboot the host after applying this workaround. In most use-case scenarios, this procedure will have a minimal effect on performance. If the affected ESXi host is non-responsive, reboot the host. Log in to the ESXi host as the root user via SSH. For more information on enabling SSH, see Using Tech Support Mode in ESXi 4.1 and ESXi 5.x (1017910). Disable the TSO Zero Copy and TxZero Copy offloading features by running these commands:esxcli system settings advanced set -i 0 -o /Net/E1000TxTsoZeroCopyesxcli system settings advanced set -i 0 -o /Net/E1000TxZeroCopy Disconnect, then reconnect the NIC to allow the changes take effect: Right-click the affected virtual machine and click Edit Settings. Click the virtual NIC and deselect Connected. Click OK. Select Connected again and click OK. Use a different virtual NIC, such as VMXNET3. For more information, see Network Virtual Machine Configuration in the vSphere 5.1 documentation.Note: This procedure does not reclaim memory lost to the leak. To recover lost memory, reboot the host after applying this workaround.

History Update

05/09/2014 - Added (Build 1612806) under Resolution.

Related Information

TCP and UDP Ports required to access VMware vCenter Server, VMware ESXi and ESX hosts, and other network componentsUsing Tech Support Mode in ESXi 4.1, ESXi 5.x, and ESXi 6.xHow to file a Support Request in Customer ConnectError message is displayed when a large number of dvPorts are in use in VMware ESXi 5.1.xVMware ESXi 5.1, Patch Release ESXi510-201402001E1000/E1000E アダプタを使用する仮想マシンのあるホストを ESXi 5.1 U2 にアップグレードするとネットワーク上の問題が発生する所含虚拟机使用 E1000/E1000E 适配器的主机在升级到 ESXi 5.1 U2 之后存在网络连接问题

Concepts

virtual machines ; log file ; host ; ESXi ; using vCloud ; traffic intermittently ; stop passing ; vmkernel ; reconnecting ; lost memory ; use-case scenarios ; offloading features ; esxcfg-vswitch