OPERATIONAL DEFECT DATABASE
...
...
Kernel heap memory used to store DFW rules and containers is exhausted in some scale-out environments where 1000 numbers of SG (Security Groups) and 1000s of IP sets are used Distributed firewall rules are reported in the User Interface (UI) as out of sync New DFW rules fail to be publishedNote: For additional symptoms and log entries, see the Additional Information section.
This is a known behavior in VMware NSX for vSphere 6.2.x where heap memory is limited to 1.5GB.Notes: In VMware NSX for vSphere 6.2.4 and later, the heap memory has been increased from 1.5GB to 3GB for Servers with 128GB or more memory. This improves consolidation ratio. The increase from 1.5GB to 3GB in NSX for vSphere 6.2.4 is only applicable for vSphere 6.0 and later. If you run NSX for vSphere 6.2.4 on vSphere 5.5, the 1.5GB limit still applies. In VMware NSX for vSphere 6.3.0, Global Address Sets is introduced which significantly reduces heap memory. Global Address Sets are disabled by default.To avoid this issue, VMware recommends you to use one of these options: Ensure that free heap memory is always greater than 20%. Configure Distributed Resource Scheduling (DRS) to Manual mode and migrate virtual machines manually to maintain the right consolidation ratio. Note: For more information on design recommendations, see the Professional Services.
You experience these additional symptoms: In the /var/log/vmkwarning.log file on the ESXi host, you see entries similar to:WARNING: Heap: 3583: Heap vsip already at its maximum size. Cannot expand. In the /var/log/vsfwd.log file on the ESXi host, you see entries similar to:INFO messagingTaskExecutor-2 SystemEventDaoImpl:133 - [SystemEvent] Time:'Wed Jun 08 14:38:00.764 AEST 2016', Severity:'Major', Event Source:'host-10696', Code:'301032', Event Message:'Failed to apply firewall rule to vnic.', Module:'vShield Firewall', Universal Object:'false'INFO messagingTaskExecutor-2 ConfigurationPublisher:229 - Updating host host-10696 status for firewall, generation -1 ; StatusCode - 301032, Status Message - eventId: 301032 -- The Status Code "StatusCode - 301032" tell that the vibs is failing to install on the respective host which is "host-10696"(company.com).vsfwd: [WARN] failed to apply ruleset: agent vmware-sfw vnic 500ab00d-ce7b-ca98-2016-5c297dd6e703.000 gennum 1464955567691vsfwd: [WARN] failed to execute command CreateAddrSet: out of memoryNote: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.DFW ルールとコンテナの格納に使用する NSX カーネル ヒープ メモリが枯渇する用于存储 DFW 规则和容器的 NSX 内核堆内存耗尽
Click on a version to see all relevant bugs
VMware Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
