Symptoms

An incorrect Java package has been installed on the REST API hosting server. All Avamar REST API calls are failing with HTTP code 404: curl -v -k -u "admin:password" -X POST https://rest-server.dell.com:8543/rest-api/login* About to connect() to rest-server.dell.com port 8543 (#0)* Trying 10.10.10.10... connected> Host: rest-server.dell.com:8543> Accept: */*>* Closing connection #0* SSLv3, TLS alert, Client hello (1): REST API server logs show the service is not fully initialized: Snip from restserver.log2018-04-20 09:32:12,337|WARN |localhost-startStop-1|org.springframework.security.config.http.MatcherType|'path-type' is deprecated. Please use 'request-matcher' instead.2018-04-20 09:32:14,163|ERROR|localhost-startStop-1|org.springframework.web.context.ContextLoader|Context initialization failedorg.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:999)2018-04-20 09:32:14,177|WARN |localhost-startStop-1|org.springframework.web.context.support.XmlWebApplicationContext|Exception thrown from ApplicationListener handling ContextClosedEventjava.lang.IllegalStateException: ApplicationEventMulticaster not initialized - call 'refresh' before multicasting events via the context: Root WebApplicationContext: startupdate [Fri Apr 20 09:32:12 CEST 2018]; root of context hierarchy at org.springframework.context.support.AbstractApplicationContext.getApplicationEventMulticaster(AbstractApplicationContext.java:347) at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:33 Tomcat server logs reports that REST API server can t be deployed (For REST API v7.5.1 check jetty.log): Sinp from Tomcat server catalina.out logApr 20, 2018 9:32:08 AM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive /usr/local/avamar-tomcat-7.0.42/webapps/rest-api-doc.warApr 20, 2018 9:32:08 AM org.apache.catalina.startup.HostConfig deployWARINFO: Deploying web application archive /usr/local/avamar-tomcat-7.0.42/webapps/rest-api.warApr 20, 2018 9:32:08 AM org.apache.catalina.loader.WebappClassLoader validateJarFileINFO: validateJarFile(/usr/local/avamar-tomcat/webapps/rest-api/WEB-INF/lib/javax.el-api-3.0.0.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/el/Expression.classApr 20, 2018 9:32:08 AM org.apache.catalina.loader.WebappClassLoader validateJarFileINFO: validateJarFile(/usr/local/avamar-tomcat/webapps/rest-api/WEB-INF/lib/servlet-api-2.5.jar) - jar not loaded. See Servlet Spec 2.3, section 9.7.2. Offending class: javax/servlet/Servlet.class[EXT] 2018-04-20 09:32:12,337|WARN |localhost-startStop-1|org.springframework.security.config.http.MatcherType|'path-type' is deprecated. Please use 'request-matcher' instead.[EXT] 2018-04-20 09:32:14,163|ERROR|localhost-startStop-1|org.springframework.web.context.ContextLoader|Context initialization failedorg.jasypt.exceptions.EncryptionOperationNotPossibleException: Encryption raised an exception. A possible cause is you are using strong encryption algorithms and you have not installed the Java Cryptography Extension (JCE) Unlimited Strength Jurisdiction Policy Files in this Java Virtual Machine at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.handleInvalidKeyException(StandardPBEByteEncryptor.java:999) at org.jasypt.encryption.pbe.StandardPBEByteEncryptor.decrypt(StandardPBEByteEncryptor.java:976) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745)Apr 20, 2018 9:32:14 AM org.apache.catalina.core.StandardContext startInternalSEVERE: Error listenerStartApr 20, 2018 9:32:14 AM org.apache.catalina.core.StandardContext startInternalSEVERE: Context [/rest-api] startup failed due to previous errors[EXT] 2018-04-20 09:32:14,177|WARN |localhost-startStop-1|org.springframework.web.context.support.XmlWebApplicationContext|Exception thrown from ApplicationListener handling Java default directory shows that Java latest points to Java JDK instead of Java JRE: ls -l /usr/java/total 163976lrwxrwxrwx 1 root root 16 Nov 26 2014 default -> /usr/java/latestdrwxr-xr-x 9 root root 4096 Apr 19 12:16 jdk1.8.0_121drwxr-xr-x 6 root root 4096 Nov 26 2014 jre1.7.0_51lrwxrwxrwx 1 root root 22 Apr 19 12:16 latest -> /usr/java/jdk1.8.0_121lrwxrwxrwx 1 root root 16 Nov 26 2014 ucasjre -> /usr/java/latest Checking installed Java packages shows Java JDK (Java Delevelopment Kit) installed on REST API hosting server (BRM v1.2.1-13 for REST API v7.1 to v7.5.0 or SLES 12 for REST API v7.5.1): rpm -qa | grep jdkjdk1.8.0_121-1.8.0_121-fcs

Cause

Installing the incorrect Java package on the REST API hosting server causes the REST API server initialization to fail. All REST API calls fail with HTTP code 404 Not Found which indicates that we were able to communicate with the REST API server, but the server could not find the requested calls. The hosting server for REST API requires Java JRE v7 or v8 only (According to REST API server and Avamar server versions).

Resolution

Log in to the REST API hosting server through an SSH session. Switch to the root user: su - Uninstall the incorrect Java JDK package: rpm -qa | grep jdk | xargs rpm -e Confirm that the package has been removed: rpm -qa | grep jdk (This command shouldn't return any output) Confirm that the Java default directory points to JRE v7 or JRE v8: with the command ls -l /usr/java/ ls -l /usr/java/total 163972lrwxrwxrwx 1 root root 16 Apr 20 10:35 default -> /usr/java/latestdrwxr-xr-x 6 root root 4096 Nov 26 2014 jre1.7.0_51lrwxrwxrwx 1 root root 21 Apr 20 10:35 latest -> /usr/java/jre1.7.0_51lrwxrwxrwx 1 root root 16 Nov 26 2014 ucasjre -> /usr/java/latest 6. Restart the REST API server: For REST API v7.1.x to v7.5.0: /usr/localavamar/bin/restserver.sh --stop && /usr/localavamar/bin/restserver.sh --start For REST API v7.5.1: systemctl restart concerto.service 7. Confirm that REST API calls are running fine, you can use a version retrieval call as shown in the example below: curl -k -D- -X GET https://rest-server.dell.com:8543/rest-api/versionsHTTP/1.1 200 OKDate: Fri, 20 Apr 2018 11:51:25 GMTContent-Type: application/xml; version=1.0Transfer-Encoding: chunkedConnection: closeServer: Avamar1.0EMC Data Protection Service Providerhttps://rest-server.dell.com:8543/rest-api/login2.0EMC Data Protection Service Providerhttps://rest-server.dell.com:8543/rest-api/login

Support Cases