Symptom

The issue occurs in the DR environment with a 3+3 cluster node setup running version 20.15.3. Controller certificates for both DC and DR clusters are getting deletedin the following scenarios: i. DR Re-registration: After performing DR re-registration, the controller certificates for the primary cluster nodes were deleted within a day. ii. DR Failover: When a failover to the DR cluster was performed, all controller certificates on the DR cluster nodes were deleted as well. iii. Upgrade Activity:During the upgrade activity, Initially, performing a DR failover from DC to the DR cluster did not trigger the certificate issue. However, after upgrading the standby cluster nodes and falling back to DC as the primary, the issue reoccurred, resulting in control connections between all controllers and edge devices going down. iv. Certificate Serial Number Mismatch: On the primary cluster nodes, there is a discrepancy between the certificate serial numbers shown in the Neo4j database and those displayed by the show control local properties command, indicating a mismatch in the control certificates. v. Missing Certificate Serial (DR Cluster): For the standby DR cluster nodes, the control certificate serial was missing and displayed as "No certificate installed." However, the Neo4j database still contained certificate serial entries for these nodes, which were not reflected in the outputs of show control local properties or show certificate install. vi. Manual Updates Required: After installing new controller certificates on both DC and DR cluster nodes, the new certificate serials were not automatically updated in the Neo4j database. Manual updates were required to synchronize the certificate serials for both clusters.

Conditions

- Performing a disaster recovery failover results in all controller certificates on the DR cluster nodes being revoked. - After installing new controller certificates on both the active and standby cluster nodes, the new certificate serials are not automatically updated in the Neo4j database table. - The controller certificate authorization is signed by the Enterprise certificate.

Workaround

- Each time the issue occurs, new controller certificates are installed. - The certificate serials in the Neo4j database are manually updated with assistance from Cisco TAC.

Further Problem Description