Symptom

vManage 3 node cluster running 20.6.2.2 the cluster fails to elect leader, hence, causing issues with GUI login. could cause other issues as well.

Conditions

vmanage-server.log when the issue started: 17-Oct-2022 18:31:31,388 AEDT ERROR [vmanage-1] [SystemDeviceDAO] (default task-40321) |default| Failed to get chained row for keys {uuid=3e541aeb-6137-4a3a-9ef3-f8899bf3fc35} org.neo4j.driver.exceptions.TransientException: Unable to get bolt address of LEADER for database 'neo4j' /var/log/nms/containers/configuration-db/debug.log 2022-10-17 07:34:44.279+0000 DEBUG [c.n.c.r.l.p.GetRoutingTableProcedureForSingleDC] Routing result for database DatabaseId{84f956d7[neo4j]} and routing context Map{address -> String("10.33.3.132:7687")} is RoutingResult{routeEndpoints=[10.33.3.133:7687], writeEndpoints=[], readEndpoints=[10.33.3.133:7687], timeToLiveMillis=150000} 2022-10-17 07:34:54.894+0000 WARN [c.n.c.c.RaftMessageDispatcher] Unable to process message {raftId: RaftId{84f956d7}, receivedAt: 2022-10-17T07:34:54.894254Z, message: PreVote.Request from MemberId{03673601} {term=55, candidate=MemberId{03673601}, lastAppended=101560746, lastLogTerm=55}} because handler for Raft ID RaftId{84f956d7} is not installed 2022-10-17 07:34:57.839+0000 DEBUG [c.n.c.r.l.DefaultLeaderService] Topology service for database DatabaseId{84f956d7[neo4j]} returned leader ID Optional.empty 2022-10-17 07:34:57.839+0000 DEBUG [c.n.c.r.l.DefaultLeaderService] Topology service for database DatabaseId{84f956d7[neo4j]} returned leader ID Optional.empty 2022-10-17 07:34:57.839+0000 DEBUG [c.n.c.r.l.DefaultLeaderService] Leader for database DatabaseId{84f956d7[neo4j]} has Bolt address Optional.empty 2022-10-17 07:34:57.839+0000 DEBUG [c.n.c.r.l.p.GetRoutingTableProcedureForSingleDC] No leader server found. This can happen during a leader switch. No write end points available

Workaround

none at the moment

Further Problem Description