Info

I created a client in Ruby configured to establish 20,000 connections to local replica set servers (PSSA), as follows: Mongo::Client.new(['localhost:34420'],min_pool_size:20000,max_pool_size:100000) At about 6,000 connections per each of the servers, the servers start closing connections. Looking in the server log I see: {"t":{"$date":"2020-05-29T21:49:15.723-04:00"},"s":"W", "c":"EXECUTOR","id":22993, "ctx":"conn5604","msg":"Terminating session due to error: {status}","attr":{"status":{"code":1,"codeName":"InternalError","errmsg":"failed to create ser vice entry worker thread"}}} This message appears to be produced in src/mongo/transport/service_entry_point_utils.cpp: } catch (...) { return {ErrorCodes::InternalError, "failed to create service entry worker thread"}; } The above code appears to discard the root cause of the error, making further troubleshooting impossible. As a MongoDB user I would like the server to provide error messages that indicate the cause of the problem, so that I can troubleshoot the problems.

Top User Comments

g.fanini@gmail.com commented on Sun, 3 Oct 2021 16:25:38 +0000: excuse me, may I ask for an information, is it thinkable that launchServiceWorkerThread is continuously spawning a thread for each command received from a client in a continuous session, at least compiled for windows not linux ? surely I misunderstood ? xgen-internal-githook commented on Tue, 7 Jul 2020 15:02:16 +0000: Author: {'name': 'Andrew Chen', 'email': 'a.chen@mongodb.com', 'username': 'AndrooTheChen'} Message: SERVER-48509 fixed uassert condition Branch: master https://github.com/mongodb/mongo/commit/f39585101d93f47c216ea8c30e276ac0410c30a2 xgen-internal-githook commented on Tue, 7 Jul 2020 15:02:10 +0000: Author: {'name': 'Andrew Chen', 'email': 'a.chen@mongodb.com', 'username': 'AndrooTheChen'} Message: SERVER-48509 More revisions to exception logging Branch: master https://github.com/mongodb/mongo/commit/3974f1e8dd738579177245d1f9d0238cb84ac81f xgen-internal-githook commented on Tue, 7 Jul 2020 15:02:03 +0000: Author: {'name': 'Andrew Chen', 'email': 'a.chen@mongodb.com', 'username': 'AndrooTheChen'} Message: SERVER-48509 Added uassert and modified catch block Branch: master https://github.com/mongodb/mongo/commit/64fa847bcb999ceb5d01e4e39b266fbacdbb00cb xgen-internal-githook commented on Tue, 7 Jul 2020 15:01:36 +0000: Author: {'name': 'Andrew Chen', 'email': 'a.chen@mongodb.com', 'username': 'AndrooTheChen'} Message: SERVER-48509 Catch and log exceptions when creating threads fail Branch: master https://github.com/mongodb/mongo/commit/0230f90f1b8e3bda4af59f4b32b5f265220f420a ben.caimano commented on Mon, 1 Jun 2020 17:50:15 +0000: oleg.pudeyev, there should also be this log statement which does specify the reason behind the failure. That said, the try-catch behavior does have a small possibility for us to swallow exceptions if they weren't thrown in those lines. I think this is separate from SERVER-47075 but maybe worth doing a touch of work: we can throw with the error statement we expect, catch a std::exception, and return a status with `e.what()` attached. We should also probably mark launchServiceWorkerThread noexcept. bruce.lucas@10gen.com commented on Sat, 30 May 2020 14:18:40 +0000: Looks like we'll be touch this log line soon as part of SERVER-47075; maybe we can fix this then. oleg.pudeyev commented on Sat, 30 May 2020 02:54:40 +0000: Test code: https://github.com/p-mongo/tests/blob/master/connect-limit/test.rb Not relevant to this ticket but connections for each server are established sequentially. The three servers are being connected to in parallel. Output: https://gist.github.com/p-mongo/aaf3a9351e46c2b06bf25f6d3b5c4ee1 It seems that when the total # of connections in the system is about 10,000, the server fails in the manner indicated. The connections could be split evenly or unevenly across the server processes. Sometimes a particular server does not fail when the total number of connections is 10,000 (possibly because it is not being connected to at that moment, when the next connection happens the number of connections has already dropped).

Steps to Reproduce