Symptoms

Recent upgrade of Avamar Server to 7.4.x or 7.5.xRecent upgrade of the vCenter server to 6.0.x or greaterRecent addition of more vCenter servers to Avamar. After upgrading to Avamar 7.4.x or Avamar 7.5.x version, MCS fails leaving a crash dump: Another common symptom for this issue is that the MCS startup process regularly fails with OutOfMemoryError. The error message is logged inside the /usr/local/avamar/var/mc/server_log/mcserver.out file. java.lang.OutOfMemoryError: GC overhead limit exceeded Dumping heap to /usr/local/avamar/var/mc/java_pid167722.hprof ... Heap dump file created [2727235681 bytes in 28.766 secs] Exception in thread "NodelistCache_thread" java.lang.OutOfMemoryError: GC overhead limit exceeded On the Avamar utility node (single node): Symptoms:Verify that MCS is down:dpnctl status mcs Identity added: /home/admin/.ssh/admin_key (/home/admin/.ssh/admin_key) dpnctl: INFO: MCS status: down. dpnctl: INFO: [see log file "/usr/local/avamar/var/log/dpnctl.log"] mcserver.sh --status === BEGIN === check.mcs (poststart) ERROR: check.mcs: [dell-emc-ave18] Attempting to connect to dell-emc-ave18:7778 CHECK FAILED: Unable to open socket ERROR: check.mcs: [dell-emc-ave18] Attempting to ping to dell-emc-ave18:28001 CHECK FAILED: Unable to open socket check.mcs failed *** FAIL *** check.mcs FAILED OVERALL (poststart) The Administrator Server is not running. ERROR: Unable to connect to the Administrator Server when it's not running. Start the Administrator Server first. Verify that hprof crash files exist:ls -hl /data01/avamar/var/mc/*.hprof -rw------- 1 admin admin 2731920459 May 16 08:41 java_pid22350.hprof -rw------- 1 admin admin 2731742194 May 16 15:32 java_pid21891.hprof -rw------- 1 admin admin 3008167674 May 18 13:18 java_pid1158.hprof -rw------- 1 admin admin 2724916194 May 18 15:39 java_pid11716.hprof -rw------- 1 admin admin 2894417851 May 19 22:20 java_pid16348.hprof -rw------- 1 admin admin 2717897589 May 29 23:56 java_pid22425.hprof -rw------- 1 admin admin 2727161261 May 31 12:13 java_pid32058.hprof -rw------- 1 admin admin 2730578140 May 31 14:14 java_pid1311.hprof -rw------- 1 admin admin 3012851350 Jun 11 11:07 java_pid17065.hprof This issue has so far been seen when there are more than 5 vCenters attached to the Avamar grid. Verify that multiple vCenters exist:mcserver.sh --status |grep "^vCenter:" vCenter: vcenter-net.dell.emc.local vCenter: vcenter-or.dell.emc.local vCenter: vcenter-test.dell.emc.local vCenter: vcenter-sp.dell.emc.local vCenter: vcenter-pr02.dell.emc.local vCenter: vcenter-vkm.dell.emc.local vCenter: vcenter-dev5.dell.emc.local vCenter: vcenter-prod01.dell.emc.local vCenter: vcenter-dls01.dell.emc.local vCenter: vcenter-pr01.dell.emc.local vCenter: vcenter-tel.dell.emc.local mcserver.sh --status |grep -c "^vCenter:" 11 "GC overhead limit exceeded" messages can be seen in the mcserver.out file. grep "Exception.*limit exceeded" /usr/local/avamar/var/mc/server_log/mcserver.out Exception in thread "VmwareEventsMonitor for VC /vcenter-ds01.dell.emc.local" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "RabbitMQ monitor thread" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "pool-18-thread-18" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "VmwareEventsMonitor for VC /vcenter-sp.dell.emc.local" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "VmwareEventsMonitor for VC /vcenter-net.dell.emc.local" java.lang.OutOfMemoryError: GC overhead limit exceeded Exception in thread "VmwareEventsMonitor for VC /vcenter-or.dell.emc.local" java.lang.OutOfMemoryError: GC overhead limit exceeded

Cause

Since vCenter 6.0, the memory requirements for the vCenter connections have been increased. When the MCS attempts to initiate parallel connections to multiple vCenters, it exhausts the javaHeap. Each vCenter connection takes 23 MB. The parameter connection_pool_size, listed in the mcserver.xml controls the number of connections that MCS makes to each vCenter. It is by default set to 5. For example, if there are 7 vCenters attached to the Avamar server, during the MCS startup, the total memory consumed is as follows: 7 (vCenters) * 5 (pool size) * 23 = 805 MB Note: The default javaHeap is 1 GB (for AVE/Single node grids) and 2 GB (for multi node grids), and MCS needs at least 0.5 GB memory for other internal processes. When the total memory used by the MCS exceeds the memory allocated memory in the javaHeap, the MCS crashes.

Resolution

Engage DELL Avamar Technical Support to correct the value for vCenter connection pools to prevent this issue from happening again.

Support Cases