Symptom

Periodic sync between a high availability (HA) pair of Cisco Secure Firewall Management Centers (FMCs) may fail in the first cycle following resuming sync. On the FMC of the HA pair that was active at the time of resuming sync, the /var/log/action_queue.log file may contain an error message containing the following error message (where [file hash] will be the MD5 hash of a file that the FMCs intend to keep in sync between the FMCs -- and this value will differ between instances of this defect): --- Caught exception in file_process from SF::Transaction::HADC::_localFile: Can't use string ("[file hash]") as a HASH ref while "strict refs" in use at /usr/local/sf/lib/perl/5.32.1/SF/Transaction/HADC.pm line 2374. --- The same error may appear multiple times for successive FMC HA periodic sync attempts.

Conditions

FMCs are in high availability. The FMCs were just upgraded to a system software version that changes two key operational aspects of FMC HA sync: * the structure of a metadata file that FMC HA sync uses for tracking the MD5 hashes and last modification times of files that the active FMC ensures are in sync on the standby FMC. * the method the active FMC uses for synchronizing config archives (artifacts from past deployments to managed devices that the FMC can use to show more details about past deployments and to roll back to a previous deployment) to the standby FMC At the time of the creation of this defect, this will include upgrades of the FMC directly to version 7.4.1 or later from any older version with the exception of coming from version 7.2.6 or later within major software version 7.2. For example, an upgrade of the FMC from 7.2.1 to 7.4.2 would encounter this defect, while an upgrade from version 7.2.9 to 7.4.1 would not encounter this defect. The "Make Me Active" option was selected on the GUI of one of the FMCs (Integration > Other Integrations > High Availability) to make that FMC active (and the other FMC standby) within the HA pair, in terms of current operating role.

Workaround

1. Log into the CLI of the active FMC. Go to expert mode. 2. Execute the following command (when prompted, enter the password of the CLI user) to delete the metadata files the active FMC uses to track the statuses of the files the FMC HA sync process keeps in sync between the FMCs: sudo rm -v /etc/sf/changed_files.json* As long as the command at this step was properly executed, it is acceptable to see "rm: cannot remove '/etc/sf/changed_files.json*': No such file or directory" if the metadata files do not exist on the FMC since the goal is to ensure that the files are not there so that the FMC can freshly generate them them during the next FMC HA periodic sync cycle. 3. Repeat steps 1-2, but on the standby FMC.

Further Problem Description