Loading...
Loading...
(Test only issue) Issue There is a potential deadlock between checkMetadataConsistency on secondaries and rollback, as follows: [ROLLBACK] A rollback starts. [CMC] A _shardsvrCheckMetadataConsistencySecondaryParticipant command arrives and starts running. [CMC] Inside checkCollectionMetadataInShardCatalog, it calls into checkCatalogStability, which acquires the CSR lock. [CMC] Still inside checkCollectionMetadataInShardCatalog with the CSR lock held, it calls into getShardCatalogRelaxedChunkInfo, which will do a snapshot read via DBDirectClient. [ROLLBACK] Calls into ShardingRecoveryService::onReplicationRollback, then ShardingRecoveryService::_resetInMemoryStates, which tries to acquire the CSR lock. At this point the checkMetadataConsistency is waiting for the rollback to finish to satisfy the read concern, and the rollback is waiting for the CSR lock, so we are deadlocked. Note Rollback kills running operations, so it is not possible to reverse steps (1) and (2) - the _shardsvrCheckMetadataConsistencySecondaryParticipant must start after rollback. In particular, with SERVER-132105, this particular deadlock doesn't seem to reproduce, since _shardsvrCheckMetadataConsistencySecondaryParticipant can't start during rollback in step (2). Fix We should immediately release the CSR lock (similar to how it's done below) to avoid calling into DBDirectClient with locks held.
Click on a version to see all relevant bugs
MongoDB Integration
Learn more about where this data comes from
BugZero Plan
Streamline upgrades with automated vendor bug scrubs
BugZero Prevent
Wish you caught this bug sooner? Get proactive today.