Symptom
FMC will display different state than what the FTD HA pair has. Failover works fine on the devices but on UI it is not showing correct status
VMS logs will show some errors like:
31-Dec-2018 08:14:39.045,[ERROR],(HANodeStatusProcessor.java:118),Skipping. Node status : CLUSTER_STATE_SLAVE, Device id : 12884902387, old timestamp : 4294972564839, new timestamp : 1546244078930,com.cisco.nm.vms.ccm.HANodeStatusProcess
or, pool-6-thread-4
(...)
31-Dec-2018 08:16:39.037,[ERROR],(NGFWHANodeRefreshExecutorService.java:299),Node-Status-Refresh:HA Node refresh operation failed as both Nodes did not send a node status message related to HA status,com.cisco.nm.vms.device.NGFWHANodeRef
reshExecutorService, pool-66-thread-1
com.cisco.nm.vms.device.exception.VmsDeviceException$FTDHA_NODE_REFRESH_FAILED_FOR_BOTH_DEVICES: Node status refresh failed
at com.cisco.nm.vms.device.NGFWHANodeRefreshExecutorService$NGFWHANodeRefreshTask.triggerFTDHAStatusRefreshOperation(NGFWHANodeRefreshExecutorService.java:247)
at com.cisco.nm.vms.device.NGFWHANodeRefreshExecutorService$NGFWHANodeRefreshTask.call(NGFWHANodeRefreshExecutorService.java:125)
at com.cisco.nm.vms.device.NGFWHANodeRefreshExecutorService$NGFWHANodeRefreshTask.call(NGFWHANodeRefreshExecutorService.java:105)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Conditions
There is a large drift in NTP time
