OPERATIONAL DEFECT DATABASE
...
...
Netflow (ESXi IPFIX) is configured in the vCenter or SWITCH IPfix is configured on NSX-T.During the NSX-T upgrade, the FROM version of ESXi is 7.0.3 or higher and the NSX-T is 3.0.x or 3.1.x (including 3.1.3.7) Host upgrade fails with the following error vLCM based upgrade : "Upgrade failed: Failed to apply Component nsx-lcp-bundle(3.1.3.7.4-7.0.19746859): an error occurred while enabling service nsx-datapath-dl" non-VLCM based upgrade: "Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004)\nCannot remove module nsxt-ipfix-19068435: module symbols in use\n\n') vibs = ['VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480'] Please refer to the log file for more details..
The issue occurs due to changes that went into the VMKAPI for the ESXi 7.0 U3. Due to these changes, the IPFIX modules receive a non-zero reference count when an unload of the module is attempted leading to failure in the unload of the IPFIX Module.
NSX-T fails to upgrade an ESXi host
This issue is resolved in NSX-T 3.2.1 and later releases.It is also fixed in 3.1.3.7.4 .
Scenario 1: During the NSX-T upgrade from 3.0.x and 3.1.x (until 3.1.3.7) AND if the version of the ESXi is 7.0.3.x. a. Unconfigure the "Netflow" or/and "SWITCH IPFIX Profiles" from the VRNI or the third-party flow collectors before the upgrade. If applicable, disable Netflow/IPFIX in both the NSX-T and VC datasources in VRNI. b. Wait for at least 5 minutes after unconfiguring IPFIX, and reboot the ESXi host. c. Attempt the NSX-T host upgrade. Scenario 2: If the user had used IPFIX (DVS/Switch) after the last reboot of the ESXi host, the NSX-T host upgrade may fail and the user has to reboot the ESXi host to get out of the error state and retry the upgrade. Scenario 3: During the NSX-T upgrade to 3.2.1 or later releases AND if the version of the ESXi is 7.0.3.x., NSX-T introduced a pre-check to alert the user about the IPFIX config and the hosts may be automatically rebooted during the upgrade without user intervention. Note: 1) As the host must be rebooted to workaround the issue, the in-place host upgrade mode cannot be used. 2) Both Netflow and/or SWITCH IPFIX profiles can be re-configured post NSX-T host upgrade. 3) Unloading IPFIX module issue does NOT manifest when just "Firewall IPFIX Profiles" is configured, hence there is NO need to clear the "Firewall IPFIX Profiles" or the Collector configuration for type "IPFIX Profile" 4)In the case of the upgrade via VLCM the precheck will only alert the user, but the user has to reboot the ESXi host manually to complete the NSX-T host upgrade. Ensure that no VM migrates to the rebooted ESXi host before the completion of host upgrade. 5) If the user has overridden Netflow policy for a specific dvport ID, ensure that Netflow is disabled for that dvport ID.
On the NSX-T UI, an error similar to the following is displayed. Install of offline bundle failed on host 0b830eb1-4782-4c90-8065-7e1606438509 with error : [LiveInstallationError] VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: /usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py:352: SyntaxWarning: "is not" with a literal. Did you mean "!="? if DEV_BUILDTYPE is not "beta": start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1154, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 953, in DualLoadUpgrade PreUpgrade() File "/etc/init.d/nsx-datapath-dl", line 794, in PreUpgrade UnloadNonDLModules() File "/etc/init.d/nsx-datapath-dl", line 146, in UnloadNonDLModules nsxesxutils.unloadModule(modName, False) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 446, in unloadModule raise Exception('Failed to unload module %s: %s' % Exception: Failed to unload module nsxt-ipfix-19068435: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004) Cannot remove module nsxt-ipfix-19068435: module symbols in use It is not safe to continue. Please reboot the host immediately to discard the unfinished update. cause = ('nsx-lcp-bundle(3.1.3.7.0-7.0.19380480)', 'nsx-datapath-dl', 'Error in running [/etc/init.d/nsx-datapath-dl start upgrade]:\nReturn code: 1\nOutput: /usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py:352: SyntaxWarning: "is not" with a literal. Did you mean "!="?\n if DEV_BUILDTYPE is not "beta":\nstart upgrade begin\nException:\nTraceback (most recent call last):\n File "/etc/init.d/nsx-datapath-dl", line 1154, in <module>\n DualLoadUpgrade()\n File "/etc/init.d/nsx-datapath-dl", line 953, in DualLoadUpgrade\n PreUpgrade()\n File "/etc/init.d/nsx-datapath-dl", line 794, in PreUpgrade\n UnloadNonDLModules()\n File "/etc/init.d/nsx-datapath-dl", line 146, in UnloadNonDLModules\n nsxesxutils.unloadModule(modName, False)\n File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 446, in unloadModule\n raise Exception(\'Failed to unload module %s: %s\' %\nException: Failed to unload module nsxt-ipfix-19068435: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004)\nCannot remove module nsxt-ipfix-19068435: module symbols in use\n\n') vibs = ['VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480'] Please refer to the log file for more details.. On the ESXi host /var/run/log/esxupdate.log following error is observed: 2022-xx-xxTxx:xx:24Z esxupdate: 2136236: LiveImageInstaller: DEBUG: Output: being upgraded Killed failed to start 2022-xx-xxTxx:xx:31Z esxupdate: 2136236: HostImage: DEBUG: installer LiveImageInstaller failed: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: VMware_bootbank_nsx-esx-datapath_3.1.3.7.0-7.0.19380480: Error in running [/etc/init.d/nsx-datapath-dl start upgrade]: Return code: 1 Output: /usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py:352: SyntaxWarning: "is not" with a literal. Did you mean "!="? if DEV_BUILDTYPE is not "beta": start upgrade begin Exception: Traceback (most recent call last): File "/etc/init.d/nsx-datapath-dl", line 1154, in <module> DualLoadUpgrade() File "/etc/init.d/nsx-datapath-dl", line 953, in DualLoadUpgrade PreUpgrade() File "/etc/init.d/nsx-datapath-dl", line 794, in PreUpgrade UnloadNonDLModules() File "/etc/init.d/nsx-datapath-dl", line 146, in UnloadNonDLModules nsxesxutils.unloadModule(modName, False) File "/usr/lib/vmware/nsx-esx-datapath/lib/python3.5/nsxesxutils.py", line 446, in unloadModule raise Exception('Failed to unload module %s: %s' 2022-03-17T04:47:31Z esxupdate: % Exception: Failed to unload module nsxt-ipfix-19068435: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004) Cannot remove module nsxt-ipfix-19068435: module symbols in use It is not safe to continue. Please reboot the host immediately to discard the unfinished update.. Clean up the installation. 2022-03-17T04:47:31Z esxupdate: tils.py", line 446, in unloadModule\n raise Exception(\'Failed to unload module %s: %s\' %\nException: Failed to unload module nsxt-ipfix-19068435: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004)\nCannot remove module nsxt-ipfix-19068435: module symbols in use\n\n\nIt is not safe to continue. Please reboot the host immediately to discard the unfinished update.']', outfile = 'None', returnoutput = 'True', timeout = '0.0'. 2022-03-17T04:47:31Z esxupdate: 2136236: root: ERROR: raise Exception('Failed to unload module %s: %s' % 2022-03-17T04:47:31Z esxupdate: 2136236: root: ERROR: Exception: Failed to unload module nsxt-ipfix-19068435: vmkmod: VMKMod_UnloadModule: Unloading module nsxt-ipfix-19068435 failed: Busy (bad0004)
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
