Symptoms
The EMT, also known as Tomcat, service shows that it is running, but port 8543 is not listening for connections.This causes issues with functions such as:
Proxy deployment managerSnapshot managerRabbitMQDTLTTopology managerConnection manager
Restarting Tomcat, as shown below, does not resolve the behavior.
emwebapp.sh --stop
emwebapp.sh --start
Port 8543 is not listening and reports "connection refused".
admin@ave:/home/admin/>: curl -v localhost:8543
* About to connect() to localhost port 8543 (#0)
* Trying ::1... Connection refused
* Trying 127.0.0.1... Connection refused
* couldn't connect to host
* Closing connection #0
curl: (7) couldn't connect to host
Cause
In the mcserver.log* files: /usr/local/avamar/var/mc/server_log/mcserver.log.0We see that the log reports connections refused to port 8543
WARNING: Failed (https://localhost:8543/deploymanager/auth/logout)(attempt 2 of 3): Connect to localhost:8543 [localhost/127.0.0.1, localhost/0:0:0:0:0:0:0:1] failed: Connection refused (Connection refused)
11/04-20:11:37.00805 [RMI TCP Connection(1204)-10.80.209.10#9026] com.avamar.mc.pdm.ProxyDeploymentManagerService.submitJsonWithRetry
The catalina.out log contains everything that is written to Tomcat's system.out and system.err/usr/local/avamar-tomcat/logs/catalina.out
The catalina.out file comes from the catalina.sh script that is conventionally used to launch Tomcat.
Whenever tomcat does not launch properly, or something is wrong with the Java implementation of tomcat, look at catalina.out
In this case, the output includes:
2022-11-08 20:41:13.866 ERROR 21323 --- [ost-startStop-1] o.s.boot.SpringApplication : Application run failed
java.io.FileNotFoundException: /home/admin/.keystore (Permission denied)
Root Cause/home/admin/.keystore ownership was root:root. In Avamar 19.4, it should be admin:admin.
Resolution
Update the tomcat keystore ownership to admin:admin
chown admin:admin /home/admin/.keystore
Restart tomcat.
emwebapp.sh --restart
Check that port 8543 is listening.
netstat -anp | grep 8543
