Info

Hi We face an issue when we do many map reduce operations on a sharded cluster. The used memory increases more and more and when it is on the limit of the host memory, the mongod process crashes with an out of memry exception. Environment: 24 shards, 80 GByte RAM each Default value for storage.wiredTiger.engineConfig.cacheSizeGB (no value set in the config) Continuous parallel running map reduce operations on different input and output collections (Java applicaton with 10 threads) We believe that the memory can be used by mongo up to the max availabe RAM.  But the mongod process should never crash. Many thank for a feedback!

Top User Comments

dmitry.agranat commented on Mon, 30 Mar 2020 12:22:05 +0000: hermann.jaun@comfone.com, I suggest we'll wait for another 2 weeks in this ticket just in case this issue occurs again. If it doesn't, we'll close this ticket and will review new data in a new ticket if it reoccurs sometime in the future. Does it make sense? hermann.jaun@gmail.com commented on Mon, 30 Mar 2020 12:14:44 +0000: Hi @Dmitry None of the server did crash after setting the storage.wiredTiger.engineConfig.cacheSizeGB to 35 GByte. I don't know if you want close this ticket and we open a new one as soon as we get the error again? Best Regards, Hermann dmitry.agranat commented on Sun, 29 Mar 2020 07:25:13 +0000: Hi hermann.jaun@comfone.com, Has the issue reoccured again since the last time you've initially reported it? If so, were you able to collect the requested information and upload it to the secure portal? Thanks, Dima dmitry.agranat commented on Fri, 13 Mar 2020 17:21:28 +0000: Hi hermann.jaun@comfone.com, The SERVER project is for bugs and feature suggestions for the MongoDB server. If you need further assistance with understanding how MongoDB manages memory, I encourage you to ask our community by posting on the MongoDB Community Forums or on Stack Overflow with the mongodb tag. In short, OS manages allocation and deallocation of process memory and its OS which decides when/if to release memory, for example, when there is a "memory pressure". I assume this is not the case in your example. As for the MongoDB cache, we keep MongoDB data in the cache because it might be needed again, and it's most efficient to retrieve it from the cache. Releasing cached data back to the OS defeats the purpose of a cache. Also, please keep in mind that apart from the WT cache (which is 50% of the total RAM by default), there is memory we consume outside the WT cache (connections, in-memory sorts, buffers for intermediate data sets and sorting in aggregate pipelines, memory for de-duplicating results with $in or $or conditions, memory for scoring full text index results, cursors, plan cache...). There is no parameter to release MongoDB memory on a running process. Thanks, Dima rui.ribeiro@comfone.com commented on Wed, 11 Mar 2020 00:16:13 +0000: Hi @Dmitry Agranat I would like also to understand how MongoDB manage the memory. Even if I am not running any operation the memory is never full released. As you can see in the graphic below from one of my shards. As you can see the memory is never full released to the initial value. Which parameters we can set, to force MongoDB to release all the memory when nothing is running? Thank you. dmitry.agranat commented on Thu, 5 Mar 2020 16:05:44 +0000: Hi hermann.jaun@comfone.com, looking at the mongod logs together with diagnostic.data is the key here to identify the root cause. If the issue happens again, please grab both and upload to the secure portal. hermann.jaun@comfone.com commented on Thu, 5 Mar 2020 10:07:12 +0000: Hi Dmitry I have uploaded the requested diagnostic date. In addition a screenshot showing the memory usage, when the crash happened on 1.3.2020 17:30 CET (UTC + 1). The server runs in UTC timezone. The Logs are unfortunately already deleted, we have too much transactions to keep them so long. But we will save the next time the respective log file. I would like to mention, that we meanwhile set on in the meantime on 3.3.2020 the parameter  storage.wiredTiger.engineConfig.cacheSizeGB to 35 GByte. It looks like ** since then the memory consumption stays much lower than before (same load). It is since 2 days quite stable at around 58 GByte. But we have to observe the behaviour for a longer period. Can this be the solution? For what is Mongo server using the RAM above the cache size? Best Rgards, Hermann dmitry.agranat commented on Wed, 4 Mar 2020 23:14:41 +0000: Hi hermann.jaun@comfone.com, Would you please archive (tar or zip) the mongod.log files and the $dbpath/diagnostic.data directory (the contents are described here) from the server in question and upload them to this support uploader location? Please also clarify the exact time and the time zone of the reported issue. Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Thank you Dima hermann.jaun@comfone.com commented on Tue, 3 Mar 2020 20:15:46 +0000: Hi Dmitry Hereafter the additional info: MongoDB server version 4.0.16 We use 1 mongod process only on each server No other applications on the server (standard Linux installation) dmitry.agranat commented on Tue, 3 Mar 2020 17:33:53 +0000: hermann.jaun@comfone.com, what MongoDB version do you use and how many mongod processes each server runs?

Steps to Reproduce