Symptoms
NSX-T Data Center in-place upgradeESXi host fails to upgradeOn the ESXi host, /var/log/esxupdate.log shows an out of memory error for nsx-t-vsip module
2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory2022-04-08T14:37:23Z esxupdate: 124631012: root: ERROR: Cannot load module nsx-esx-70u1/nsxt-vsip-19380480: Out of memory2022-04-08T14:37:22Z esxupdate: 124631012: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1154, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 979, in DualLoadUpgrade LoadKernelModules() File "/etc/init.d/nsx-datapath-dl", line 209, in LoadKernelModules nsxesxutils.loadModule(modName, modParam) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 462, in loadModule (moduleName, out.decode())) Exception: Failed to load module nsx-esx-70u1/nsxt-vsip-19380480: vmkmod: VMKModLoad: VMKernel_LoadKernelModule(nsxt-vsip-19380480): Out of memory
Cause
In-place upgrade of ESXi hosts allows hosts to be upgraded without the necessity of entering maintenance mode.Since the ESXi host has to continue running the current NSX version while loading the upgraded version, it requires more memory than a standard upgrade.During this dual load process for the the DFW vsip module, if the ESXi host cannot allocate sufficient memory the upgrade will fail.
Resolution
This is a known issue impacting NSX-T Data Center.
Workaround
If a host upgrade failure has already occurred, follow these steps to recover. 1. On the ESXi host that has failed to upgrade, clear the upgrade status flag
#nsxcli -c set host-switch upgrade-status false
2. Reboot the host and resume the upgrade
