...
Restoring the config-db on 20.6.2.
Restoring the config-db on 20.6.2.
We manually generated a new encrypted password and updated in the /.vmanage_service_user.json file. Cubic_vManage:/opt/web-app/etc# java -jar /opt/web-app/etc/vmanage-runutils.jar EncryptPassword 9OEM@d5whpsovZBlqCrW WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance. $CRYPT_CLUSTER$7/2HU48YhugEfXz91Rk94g==$mTbZdcwWoir5CGBP8eh9q7JzgaLf9d5FlTuuwMuurps= Updated this new encryption in /opt/web-app/etc/.vmanage_service_user.json for neo4j. Restarted the app-server after this and GUI came up.
We changed config-db credentials on 20.3.4.1 and later performed upgrade from 20.3.4.1 to 20.6.2. GUI came up fine. After this we tried to restore customer's config db on 20.6.2 where we faced issue of GUI not coming up. With the help of Charles and Yingfang we saw the password keystore was corrupted after the restore. We manually generated a new encrypted password and updated in the /.vmanage_service_user.json file. 25-Feb-2022 18:58:53,388 UTC INFO [] [CuratorClient] (vManage-akka.actor.default-dispatcher-4) || createClient: client null, url localhost:2181, isAuth true 25-Feb-2022 18:58:53,405 UTC INFO [] [CuratorFrameworkImpl] (vManage-akka.actor.default-dispatcher-4) || Starting 25-Feb-2022 18:58:53,426 UTC INFO [] [CuratorFrameworkImpl] (vManage-akka.actor.default-dispatcher-4) || Default schema 25-Feb-2022 18:58:53,451 UTC ERROR [] [ConnectionState] (vManage-akka.actor.default-dispatcher-4-EventThread) || Authentication failed 25-Feb-2022 18:58:53,456 UTC INFO [] [ConnectionStateManager] (vManage-akka.actor.default-dispatcher-4-EventThread) || State change: CONNECTED 25-Feb-2022 18:58:53,461 UTC INFO [] [EnsembleTracker] (vManage-akka.actor.default-dispatcher-4-EventThread) || New config event received: {} 25-Feb-2022 18:58:53,461 UTC INFO [] [EnsembleTracker] (vManage-akka.actor.default-dispatcher-4-EventThread) || New config event received: {} 25-Feb-2022 18:58:54,162 UTC INFO [] [EncryptionFacility] (vManage-akka.actor.default-dispatcher-4) || CryptoKey store initialized 25-Feb-2022 18:58:54,164 UTC ERROR [] [Neo4JUtil] (vManage-akka.actor.default-dispatcher-4) || Failed to decrypt the password 25-Feb-2022 18:58:54,548 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: The client is unauthorized due to authentication failure. 25-Feb-2022 18:58:54,764 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:54,764 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Failed to discover any available neo4j servers 25-Feb-2022 18:58:54,779 UTC ERROR [] [Neo4JUtil] (vManage-akka.actor.default-dispatcher-4) || Failed to decrypt the password 25-Feb-2022 18:58:54,991 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:55,202 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:55,202 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Failed to discover any available neo4j servers 25-Feb-2022 18:58:55,203 UTC ERROR [] [Neo4JUtil] (vManage-akka.actor.default-dispatcher-4) || Failed to decrypt the password 25-Feb-2022 18:58:55,414 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:55,625 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:55,625 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Failed to discover any available neo4j servers 25-Feb-2022 18:58:55,626 UTC ERROR [] [Neo4JUtil] (vManage-akka.actor.default-dispatcher-4) || Failed to decrypt the password 25-Feb-2022 18:58:55,838 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:56,050 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Error connecting to Driver: Failed to discover an available server 25-Feb-2022 18:58:56,050 UTC ERROR [] [Neo4JDriverCreator] (vManage-akka.actor.default-dispatcher-4) || Failed to discover any available neo4j servers 25-Feb-2022 18:58:56,063 UTC INFO [] [Neo4JDatabaseSetup] (vManage-akka.actor.default-dispatcher-4) || Waiting to acquire inter process lock for DB: vmanagedb, tenant: default 25-Feb-2022 18:58:56,097 UTC INFO [] [Neo4JDatabaseSetup] (vManage-akka.actor.default-dispatcher-4) || Acquired inter process lock for DB: vmanagedb, tenant: default ? true 25-Feb-2022 18:58:56,100 UTC INFO [] [ApplicationActor] (vManage-akka.actor.default-dispatcher-4) || Exception in init thread com.viptela.vmanage.server.datastore.DatabaseSetupException: Failed to create/initialize database : vmanagedb