Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
The NSX-T UI becomes unavailable.The NSX-T UI may show "Some appliance components are not functioning properly. Component health: POLICY:UNKNOWN, MANAGER:UNKNOWN, SEARCH:UNKNOWN, NO."On vSphere, it's not possible to change port groups, vMotion or power on NSX-backed VMs.The compactor throws OOM when checkpointing table LoginLogoutEvent as seen in /var/log/corfu/corfu-compactor-audit.log: 2021-10-01T10:30:16.418Z INFO main FrameworkCorfuCompactor - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfu-compactor"] Serial checkpoint for map nsx-manager LoginLogoutEvent 7103, id 3e6e9124-c492-3f2c-a0d8-58ac1bcebd3d.2021-10-01T10:30:16.418Z INFO main MultiCheckpointWriter - appendCheckpoints: appending checkpoints for 1 maps2021-10-01T10:30:16.438Z INFO main CheckpointWriter - appendCheckpoint: Started checkpoint for 3e6e9124-c492-3f2c-a0d8-58ac1bcebd3d at snapshot Token(epoch=3968, sequence=2173643076)2021-10-01T10:30:23.993Z INFO main ObjectSerializer - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfudb"] Identifier deserialization cache stat. Total=1811683, hits=40000...2021-10-01T10:38:16.093Z INFO main ObjectSerializer - - [nsx@6876 comp="nsx-manager" level="INFO" subcomp="corfudb"] Identifier deserialization cache stat. Total=11891683, hits=5080000java.lang.OutOfMemoryError: Java heap spaceDumping heap to /image/core/java_pid3360.hprof ...Heap dump file created [2718867198 bytes in 13.856 secs]Aborting due to java.lang.OutOfMemoryError: Java heap space## A fatal error has been detected by the Java Runtime Environment:## Internal Error (debug.cpp:308), pid=3360, tid=0x000074996e1ed700# fatal error: OutOfMemory encountered: Java heap space## JRE version: Java(TM) SE Runtime Environment (8.0_301-b09) (build 1.8.0_301-b09)# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.301-b09 mixed mode linux-amd64 compressed oops)# Core dump written. Default location: //core or core.3360## An error report file with more information is saved as:# //hs_err_pid3360.log## If you would like to submit a bug report, please visit:# http://bugreport.java.com/bugreport/crash.jsp#Aborted (core dumped) By the time, the compactor throws OutOfMemory, the table has a high number of entries and the table size has been growing and seems there is no cleanup task happening. 2021-10-01T00:19:21.809Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for 3e6e9124-c492-3f2c-a0d8-58ac1bcebd3d, entries(3797270), cpSize(2354490943) bytes at snapshot Token(epoch=3968, sequence=2171963962) in 1177056 ms...2021-10-01T07:10:02.856Z INFO main CheckpointWriter - appendCheckpoint: completed checkpoint for 3e6e9124-c492-3f2c-a0d8-58ac1bcebd3d, entries(4761324), cpSize(2950812316) bytes at snapshot Token(epoch=3968, sequence=2172928084) in 5079209 ms <= last successful run
No cleanup task is happening for the table LoginLogoutEvent which results in an increase in the table size. Hence, the /image/core/*.hprof files are created.
This is resolved in NSX-T version 3.1.3.3 available in VMware Downloads.
To work around this issue, contact VMware Support and note this Article ID in the problem description.