Info

Doesn't impact clusters that have the _SecondaryThrottle turned off or have the balancer off. MigrationBatchFetcher's  destructor depends on calling inserterWorkerThreadPool shutdown/join to drain all the scheduled jobs and join on the threadPool spawns another extra thread to drain it. Also, secondaryThrottle is disabled for a chunkMigrationConcurrency of more than 1 because of how sessions are checked out/in. When the MigrationBatchFetcher calls join on the inserterThreadPool and there are more jobs to process in the pool, it can spawn an extra thread to drain the pool. So now, although the threadCnt is 1 here, 2 threads are competing for secondaryThrottle and the invariant is caused. Invariant failure string in the log file looked like - {"t":\{"$date":"2023-02-28T19:58:58.693+00:00"} ,"s":"F", "c":"-", "id":23079, "ctx":"ChunkMigrationInserters-1","msg":"Invariant failure","attr":{"expr":"checkedOutSession","file":"src/mongo/db/session_catalog.cpp","line":302}} Workaround Either one of the following would work 1. Turn the balancer off 2. Turn the secondaryThrottle off(with the balancer still running).

Top User Comments

xgen-internal-githook commented on Mon, 20 Mar 2023 15:50:46 +0000: Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'} Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread (cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20) Branch: v6.0 https://github.com/mongodb/mongo/commit/033418d940372570b12ec4233946e9675fc05226 xgen-internal-githook commented on Tue, 7 Mar 2023 21:18:25 +0000: Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'} Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread (cherry picked from commit bf4b3ae9b3eb5c376201135155ef6738bfef0e20) Branch: v5.0 https://github.com/mongodb/mongo/commit/84a03cb20852a2dc0247481f7da7a023688bd146 xgen-internal-githook commented on Mon, 6 Mar 2023 21:54:43 +0000: Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'} Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread (cherry picked from commit 00dc7293c5d815e2104b5fe0c9a225332af3dfec) Branch: v6.3 https://github.com/mongodb/mongo/commit/bf4b3ae9b3eb5c376201135155ef6738bfef0e20 xgen-internal-githook commented on Mon, 6 Mar 2023 18:33:26 +0000: Author: {'name': 'Randolph Tan', 'email': 'randolph@10gen.com', 'username': 'renctan'} Message: SERVER-74501 Fix MigrationBatchFetcher/Inserter completion reliance to not spawn an extra cleanup thread Branch: master https://github.com/mongodb/mongo/commit/00dc7293c5d815e2104b5fe0c9a225332af3dfec

Steps to Reproduce