Symptom
The Secure Firewall Management Center (FMC) suddenly stops sending events to eStreamer clients.
On the FMC, the SymmetricDS (Symmetric) sync channel for syncing tables from Sybase to MariaDB (sybase_to_mysql) will have at least one batch with an error (ER) state. This will cause the syncing of the table responsible for the eStreamer event settings for eStreamer clients -- domain_control_info -- to not sync.
For the given FMC domain (will be "Global" when not using multiple FMC domains), the entry in the domain_control_info table in MariaDB will have a value of 0 for the "estreamer_events" field, while the same field in the Sybase version of the table will have a correct value. This is the result of the sync of the domain_control_info table not working.
Where Symmetric has a sync batch with an error -- which will cause other sync batches behind it to not run -- the Symmetric logs (/var/opt/symmetric/logs/symmetric.log) may show entries that look like the following (examples below, with some parts redacted):
---
[timestamp] INFO [ims-000] [DefaultDatabaseWriter] [qtp1121172875-58400] Failed to process a insert event in batch [batch_id].
[timestamp] ERROR [ims-000] [DataLoaderService] [qtp1121172875-58400] Failed to load batch [batch_id] StackTraceKey [SqlException:[exception number]]
[timestamp] ERROR [csm-001] [AcknowledgeService] [csm-001-push-1] The outgoing batch 000-[batch_id] failed. [reason]
[timestamp] INFO [csm-001] [PushService] [csm-001-push-1] There was a failure while pushing data to ims:000:000. [number] data and [number] batches were processed
---
Even more generalized errors with the operation of Symmetric may be present, and these could also prevent the domain_control_info table from syncing.
Conditions
The upgrade of the FMC to version 7.2.x directly from a pre-7.2 version.
The configuration and integration of an eStreamer client with the FMC prior to the upgrade of the FMC to version 7.2.x.
Workaround
Ask Cisco TAC for a procedure to repair database configuration of the event types that eStreamer has been configured to send to eStreamer clients and to further diagnose and remediate any issues with Symmetric (as these issues will vary).
