Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Environment:Avamar server version 7.4 or 7.5One or more of the following symptoms observed: Consistently high CPU usage on utility node, caused by java process, over 300% in top output.Avamar administrator server client is unable to connect or it is extremely slow to responsedpnctl status mcs shows mcs service is down but no one has shut down MCS.mcserver java process is not running. Listing process via 'ps -ef | grep mcserver' on utility node does not show mcserver.jar process. You can see OutOfMemoryError in /usr/local/avamar/var/mc/server_log/mcserver.out before you restart MCS: grep OutOfMemoryError /usr/local/avamar/var/mc/server_log/mcserver.out java.lang.OutOfMemoryError: GC overhead limit exceeded Verify that hprof crash files exist: ls -hl /data01/avamar/var/mc/*.hprof -rw------- 1 admin admin 2727161261 May 31 12:13 java_pid32058.hprof -rw------- 1 admin admin 2730578140 May 31 14:14 java_pid1311.hprof -rw------- 1 admin admin 2725283523 Jul 13 09:10 java_pid12716.hprof If there are more than 5 vCenter servers attached to the Avamar, stop following this KB. Please refer to KB502420
When there is a vCenter event, it triggers MCS to create a connection to vCenter. If vCenter generates a lot of events in a short time, MCS might create too many connections to vCenter. As a result, it exhausts the javaHeap.
To confirm it is caused by too many vCenter connections at a time, perform the following steps:Identify when MCS crashes, for example: ls -hl /data01/avamar/var/mc/*.hprof | tail -1 -rw------- 1 admin admin 2725283523 Jul 14 00:23 java_pid12716.hprof In the example above, since the java crash dump was generated on Jul 14 00:23, the issue should have happened before that time.Run the following command to confirm the mcserver.log during MCS crash is still available, for example: ls -ltr /usr/local/avamar/var/mc/server_log/mcserver.log* -rw------- 1 admin admin 10486005 Jul 13 22:21 mcserver.log.7 -rw------- 1 admin admin 10485994 Jul 14 02:01 mcserver.log.6 -rw------- 1 admin admin 10485938 Jul 14 04:21 mcserver.log.5 -rw------- 1 admin admin 10485831 Jul 14 06:54 mcserver.log.4 -rw------- 1 admin admin 10485810 Jul 14 09:48 mcserver.log.3 -rw------- 1 admin admin 10485877 Jul 14 13:54 mcserver.log.2 -rw------- 1 admin admin 10485913 Jul 14 18:14 mcserver.log.1 In the example above, mcserver.log.6 is likely to contain the information during MCS crash as it covers log entries between Jul 13 22:21 to Jul 14 02:01.Run the following command to check how many open vCenter connections in mcserver.log.6: grep "VCenterConnectionPool.getConnection\|VCenterConnectionPool.freeConnection" /usr/local/avamar/var/mc/server_log/mcserver.log.6 07/13-23:53:33.00933 [pool-97-thread-13#192356] com.avamar.mc.vmware.VCenterConnectionPool.freeConnection 07/13-23:53:33.00941 [pool-95-thread-54#192337] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-34#192317] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-31#192314] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-30#192313] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-29#192312] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-25#192308] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-15#192298] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-17#192300] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-49#192332] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-36#192319] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-40#192323] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-19#192302] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-21#192304] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-18#192301] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-16#192299] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-27#192310] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-28#192311] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-14#192297] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-20#192303] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00942 [pool-95-thread-33#192316] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-35#192318] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-37#192320] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-46#192329] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-50#192333] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-38#192321] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-42#192325] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-13#192296] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-12#192295] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-45#192328] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-39#192322] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-41#192324] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-47#192330] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-48#192331] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00943 [pool-95-thread-44#192327] com.avamar.mc.vmware.VCenterConnectionPool.getConnection 07/13-23:53:33.00973 [pool-97-thread-21#192364] com.avamar.mc.vmware.VCenterConnectionPool.freeConnection In the example above, we can see that:MCS created more than 30 connections from the "com.avamar.mc.vmware.VCenterConnectionPool.getConnection" messages in less than 1 second. And almost no connections freed by comparison.Confirm the number of vCenters in MCS by the following command: mcserver.sh --status |grep "^vCenter:" vCenter: vcenter-net.dell.emc.local vCenter: vcenter-or.dell.emc.local Contact Dell EMC support and tune MCS parameter or apply hotfix.