Loading...
Loading...
GUI become unreachable on all nodes 1 after another.
GUI timeout and we see errors in app-server.log: 09-Jul-2024 00:57:30,589 UTC ERROR [InvocationContextInterceptor] (timeout-thread--p11-t1) ISPN000136: Error executing command PutKeyValueCommand on Cache 'vmanage.war', writing keys [SessionCreationMetaDataKey(j1mhtCml0aLP_tIx0ljwqoCeEYSI6PIbzxAijue9)]: org.infinispan.util.concurrent.TimeoutException: ISPN000476: Timed out waiting for responses for request 24779792 from 6460fda5-0933-4b62-8722-4736813e5d0c at org.infinispan@9.4.18.Final//org.infinispan.remoting.transport.impl.MultiTargetRequest.onTimeout(MultiTargetRequest.java:167) at org.infinispan@9.4.18.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:87) at org.infinispan@9.4.18.Final//org.infinispan.remoting.transport.AbstractRequest.call(AbstractRequest.java:22) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834)
Issue started on 20.9.3 code
Restart the app-server on all the nodes.
We see this files keep growing: ls -l /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat -rw-r--r-- 1 vmanage vmanage 317861521 Jul 9 07:02 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat This file should be ideally be of 0 byes. Output collected after app-server restart: vmanage:~# while true; do date; ls -lrth /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat; sleep 300; done Tue Jul 9 20:13:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 705K Jul 9 20:13 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:18:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 919K Jul 9 20:18 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:23:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 939K Jul 9 20:23 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:28:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 939K Jul 9 20:28 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:33:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 941K Jul 9 20:33 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:38:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 1.2M Jul 9 20:38 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:43:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 1.2M Jul 9 20:43 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:48:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 1.2M Jul 9 20:48 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:53:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 1.2M Jul 9 20:53 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat Tue Jul 9 20:58:32 UTC 2024 -rw-r--r-- 1 vmanage vmanage 1.4M Jul 9 20:58 /var/lib/wildfly/standalone/data/infinispan/web/vmanage.war.dat ^C Summary: Heap dump of VM2 is token in problem state, we mainly analyze it. There are 288 restAPI threads stuck in 'com.viptela.vmanage.server.sso.saml.SAMLLoginServlet.doGet', and blocked by 'org.infinispan.cache.impl.CacheImpl.executeCommandAndCommitIfNeeded. And we have infinispan cache timeout exception in app-server log, so we suspected the issue is caused by session cache. In heap dump of vm2, there are 826,209 infinispan sessions which is unexpected. The large file size of vmanage.war.dat also confirm this. The vmanage.war.dat should be empty on health system. There should be only few session in whole cluster. But, codes of some API may create un-necessary sessions. And restAPI access of customer is very aggressive, hundred of thousands in past 2 days. 185763 /dataservice/device/omp/peers 185862 /dataservice/device/app-route/statistics 186584 /dataservice/device/control/synced/connections 187061 /dataservice/device/bfd/sessions Workaround: Restarting app-server of all nodes of the cluster may clear all the old session cache.
Cisco Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.