Info

Hi All, Today around 1 AM (GMT +2) the primary mongo crashed and other 3 mongo instances out of 5 we have in our replica set, failed successively. We are using mongodb-org-server-3.0.8-1.amzn1.x86_64 with wiredTiger engine The reason: It indicates we were hitting the 20k session limit, we understand that it might be due to not properly closing cursors but it seems to happen too often and cause the primary to shutdown instead of reject the new sessions. I start all the servers again and this is the number I had after the 4th mongo was up, db.serverStatus().wiredTiger.session { "open cursor count" : 124618, "open session count" : 17609 } When I tried to load the 5th one it failed for exceeding the limit of 20010 sessions It was obvious that there were too many open sessions, in order to close them manually. I decided to set the timeout to 5Sec by running db.runCommand({setParameter:1, cursorTimeoutMillis: 5000}) { "was" : 600000, "ok" : 1 } It seems that the default was 600 sec! After doing it and 5 sec later I run the command again and found that number of sessions decreased to 357 { "open cursor count" : 15585, "open session count" : 357 } I upload the last mongo and everything looks fine now. It is same behavior as described by Glen Miner at SERVER-17364 I attached the all information of our server (output of mdiag.sh) and snippet of the log from time of the crash. meanwhile I set the default cursor time our to 30 sec just to be on the safe side even though 5 sec seems OK as well.

Top User Comments

ramon.fernandez commented on Sun, 6 Mar 2016 14:21:05 +0000: Here's how to add this parameter to a config file: storage: dbPath: db wiredTiger: engineConfig: configString : "session_max=40000" As for your second comment, at the moment MongoDB does not have the admission control features you describe. Implementing these features is a difficult challenge, bug feel free to open a new feature request ticket. Note also that the changes in SERVER-20409 should help with this issue, but you'll need to upgrade to MongoDB 3.2.3 or newer to benefit from those changes. Regards, Ramón. dror@amobee.com commented on Fri, 19 Feb 2016 23:54:44 +0000: Hi Ramon, Two comments: 1. the --wiredTigerEngineConfigString="session_max=40000" can be set as mongod command line parameter, but I couldn't find a way to set it through the configuration file. Is it possible to configure it through the configuration file? 2. As a result of this case we intend to go over all our code and make sure any cursor is closed. But the main issue here is that a developer mistake can cause your critical path database to crush and the DB engine does not protect itself from crushing. I think the desired behavior would be that the driver (we are using the Java driver) or the server itself will reject new sessions when the limit is reached instead of opening new sessions until it get memory allocation issue and crush. Tx ramon.fernandez commented on Fri, 19 Feb 2016 12:35:22 +0000: dror@amobee.com, there's a default limit of 20000 sessions in WiredTiger. You may want to review your application to make sure unused cursors are closed to avoid consuming memory (SERVER-17364) and reaching the session limit. One alternative, as you've found out, is to lower the default cursor timeout (documented here). Another option is to increase the maximum number of sessions: --wiredTigerEngineConfigString="session_max=40000" Note however that each open session consumes memory, so it's good practice to close unused cursors. Regards, Ramón.

Steps to Reproduce