Symptoms

Unable to use Proxy Deployment Manager (PDM) to create recommendations in the Avamar User Interface (AUI) or Avamar Administrator console.

Cause

Root Causecom.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. There is an authentication issue with RabbitMQ.The 404 error code can be seen in /usr/local/avamar/var/mc/server_log/mcserver.log.0 when using PDM using Avamar Administrator console: WARNING: Failed (https://localhost:8543/deploymanager/auth/login)(attempt 1 of 3): Failed with HTTP error code : 404 additional information : HTTP Status 404 ▒~@~S Not Foundbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 404 ▒~@~S Not FoundType Status ReportMessage The requested resource [&#47;deploymanager&#47;auth&#47;login] is not availableDescription The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.Apache Tomcat 08/25-16:53:06.00956 [MCS Counter monitor thread#168] com.avamar.mc.mon.MCSPerformanceMonitor.monitor FINE: enter 08/25-16:53:06.00975 [RMI TCP Connection(122)-192.168.33.10#1523] com.avamar.mc.pdm.ProxyDeploymentManagerService.submitJsonWithRetry WARNING: Failed (https://localhost:8543/deploymanager/auth/login)(attempt 2 of 3): Failed with HTTP error code : 404 additional information : HTTP Status 404 ▒~@~S Not Foundbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 404 ▒~@~S Not FoundType Status ReportMessage The requested resource [&#47;deploymanager&#47;auth&#47;login] is not availableDescription The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.Apache Tomcat 08/25-16:53:07.00035 [RMI TCP Connection(122)-192.168.33.10#1523] com.avamar.mc.pdm.ProxyDeploymentManagerService.submitJsonWithRetry WARNING: Failed (https://localhost:8543/deploymanager/auth/login)(attempt 3 of 3): Failed with HTTP error code : 404 additional information : HTTP Status 404 ▒~@~S Not Foundbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 404 ▒~@~S Not FoundType Status ReportMessage The requested resource [&#47;deploymanager&#47;auth&#47;login] is not availableDescription The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.Apache Tomcat 08/25-16:53:07.00036 [RMI TCP Connection(122)-192.168.33.10#1523] com.avamar.mc.util.MCException.logException WARNING: com.avamar.asn.service.ServiceException: Failed with HTTP error code : 404 additional information : HTTP Status 404 ▒~@~S Not Foundbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 404 ▒~@~S Not FoundType Status ReportMessage The requested resource [&#47;deploymanager&#47;auth&#47;login] is not availableDescription The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.Apache Tomcat at com.avamar.mc.pdm.ProxyDeploymentManagerService.login(ProxyDeploymentManagerService.java:577) at com.avamar.mc.pdm.ProxyDeploymentManagerService.getAllTasks(ProxyDeploymentManagerService.java:377) at sun.reflect.GeneratedMethodAccessor95.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:318) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.security.access.intercept.aopalliance.MethodSecurityInterceptor.invoke(MethodSecurityInterceptor.java:61) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202) at com.sun.proxy.$Proxy96.getAllTasks(Unknown Source) at com.avamar.mc.pdm.ProxyDeploymentManagerServiceContext.getAllTasks(ProxyDeploymentManagerServiceContext.java:69) at sun.reflect.GeneratedMethodAccessor94.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at sun.rmi.transport.Transport$1.run(Unknown Source) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Unknown Source) at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source) Confirm the 404 error at the following endpoint: admin@ave:/usr/local/avamar/var/mc/server_log/>: curl -kv https://localhost:8543/deploymanager/auth/login * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8543 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=CA; L=Irvine; OU=Dell EMC; O=Dell Technologies; CN=ave.lab * start date: Jul 1 12:02:13 2022 GMT * expire date: Jun 28 12:02:13 2032 GMT * issuer: C=US; ST=CA; L=Irvine; OU=Dell EMC; O=Dell Technologies; CN=ave.lab * SSL certificate verify result: self signed certificate (18), continuing anyway. > GET /deploymanager/auth/login HTTP/1.1 > Host: localhost:8543 > User-Agent: curl/7.60.0 > Accept: */* > < HTTP/1.1 404 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Content-Length: 776 < Date: Thu, 25 Aug 2022 14:54:36 GMT < Connection: close < Server: Avamar < * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): HTTP Status 404 – Not Foundbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 404 – Not FoundType Status ReportMessage The requested resource [&#47;deploymanager&#47;auth&#47;login] is not availableDescription The origin server did not find a current representation for the target resource or is not willing to disclose that one exists.Apache Tomcat The snapshot manager endpoint shows the following exception stack: root@ave:/usr/local/avamar/var/mc/server_data/prefs/#: curl -kv https://localhost:8543/snapshotmanager/about * Trying ::1... * TCP_NODELAY set * Connected to localhost (::1) port 8543 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH * TLSv1.2 (OUT), TLS handshake, Client hello (1): * TLSv1.2 (IN), TLS handshake, Server hello (2): * TLSv1.2 (IN), TLS handshake, Certificate (11): * TLSv1.2 (IN), TLS handshake, Server key exchange (12): * TLSv1.2 (IN), TLS handshake, Server finished (14): * TLSv1.2 (OUT), TLS handshake, Client key exchange (16): * TLSv1.2 (OUT), TLS change cipher, Client hello (1): * TLSv1.2 (OUT), TLS handshake, Finished (20): * TLSv1.2 (IN), TLS change cipher, Client hello (1): * TLSv1.2 (IN), TLS handshake, Finished (20): * SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384 * ALPN, server did not agree to a protocol * Server certificate: * subject: C=US; ST=CA; L=Irvine; OU=Dell EMC; O=Dell Technologies; CN=ave.lab * start date: Jul 1 12:02:13 2022 GMT * expire date: Jun 28 12:02:13 2032 GMT * issuer: C=US; ST=CA; L=Irvine; OU=Dell EMC; O=Dell Technologies; CN=ave.lab * SSL certificate verify result: self signed certificate (18), continuing anyway. > GET /snapshotmanager/about HTTP/1.1 > Host: localhost:8543 > User-Agent: curl/7.60.0 > Accept: */* > < HTTP/1.1 500 < Content-Type: text/html;charset=utf-8 < Content-Language: en < Transfer-Encoding: chunked < Date: Thu, 25 Aug 2022 15:04:43 GMT < Connection: close < Server: Avamar < HTTP Status 500 – Internal Server Errorbody {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}HTTP Status 500 – Internal Server ErrorType Exception ReportMessage Servlet.init() for servlet [snapshotmanager] threw exceptionDescription The server encountered an unexpected condition that prevented it from fulfilling the request.Exceptionjavax.servlet.ServletException: Servlet.init() for servlet [snapshotmanager] threw exception org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:543) org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81) [...] java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name &#39;snapshotManagerServerConfiguration&#39;: Unsatisfied dependency expressed through field &#39;snapshotRequestHandler&#39;; nested exception is org.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name &#39;snapshotHandler&#39;: Unsatisfied dependency expressed through field &#39;snapshotTemplate&#39;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;getRestTemplate&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitTemplate]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;getRestTemplate&#39; threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;replySnapshotQueue&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598) org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:90) [...] java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.factory.UnsatisfiedDependencyException: Error creating bean with name &#39;snapshotHandler&#39;: Unsatisfied dependency expressed through field &#39;snapshotTemplate&#39;; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;getRestTemplate&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitTemplate]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;getRestTemplate&#39; threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;replySnapshotQueue&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:598) [...] java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;getRestTemplate&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitTemplate]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;getRestTemplate&#39; threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;replySnapshotQueue&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:627) [...] java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.rabbit.core.RabbitTemplate]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;getRestTemplate&#39; threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;replySnapshotQueue&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) [...] java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.factory.BeanCreationException: Error creating bean with name &#39;replySnapshotQueue&#39; defined in com.emc.avamar.vmware.snapshot.config.server.SnapshotManagerServerConfiguration: Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.support.ConstructorResolver.instantiate(ConstructorResolver.java:627) org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:456) [...] java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.beans.BeanInstantiationException: Failed to instantiate [org.springframework.amqp.core.Queue]: Circular reference involving containing bean &#39;snapshotManagerServerConfiguration&#39; - consider declaring the factory method as static for independence from its containing instance. Factory method &#39;replySnapshotQueue&#39; threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:185) [...] java.lang.Thread.run(Unknown Source) Root Causeorg.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. org.springframework.amqp.rabbit.support.RabbitExceptionTranslator.convertRabbitAccessException(RabbitExceptionTranslator.java:61) [...] org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Unknown Source) Root Causecom.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile. com.rabbitmq.client.impl.AMQConnection.start(AMQConnection.java:339) com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:647) com.rabbitmq.client.ConnectionFactory.newConnection(ConnectionFactory.java:694) [...] java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source) java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) java.lang.Thread.run(Unknown Source) * Closing connection 0 * TLSv1.2 (OUT), TLS alert, Client hello (1): Note The full stack trace of the root cause is available in the server logs.Apache Tomcat From the output above, collect the following snippet mentioned previously: threw exception; nested exception is org.springframework.amqp.AmqpAuthenticationException: com.rabbitmq.client.AuthenticationFailureException: ACCESS_REFUSED - Login was refused using authentication mechanism PLAIN. For details see the broker logfile.

Resolution

Avamar versions 19.x onwards:1. Take a backup of mcserver.xml and update the RabbitMQ password in mcserver.xml to the default 'Avamar_Rabbit' and edit the file: cp -p /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml /usr/local/avamar/var/mc/server_data/prefs/x-mcserver.xml vi /usr/local/avamar/var/mc/server_data/prefs/mcserver.xml 2. Change the 3. Restart the Management Console Service (MCS), as admin user: mcserver.sh --restart --force 4. Re-create the RabbitMQ User/Permissions/Password/vHost as root user, and use the following steps to do so automatically. service rabbitmq-server stop cd /data01/avamar/var/rabbitmq/mnesia mkdir /data01/rabbit_mnesia_backup mv rabbit@localhost* /data01/rabbit_mnesia_backup/ ls -ltr /data01/rabbit_mnesia_backup/ service rabbitmq-server start msgbrokerctl.pl --remove msgbrokerctl.pl --stop msgbrokerctl.pl --start 5. Restart Tomcat as root user. emwebapp.sh --restart 6. Test PDM recommendation.

Support Cases