Info

The sweep server under some conditions will evict entire b-trees, and for a large b-tree this can take an extended time (many minutes in some cases). While this is occurring any attempt to access the b-tree hangs. Example: start mongod with --storageEngine wiredTiger --wiredTigerCacheSizeGB 9. Then populate a collection with 8 GB of collection and index data (using small documents as below probably exacerbates the issue because it makes the eviction slower due to the number of buffers that must be freed during the eviction): for t in $(seq 16); do mongo test --eval " every = 10000 for (var i=0; ; ) { var bulk = db.c.initializeUnorderedBulkOp(); for (var j=0; j<every; j++, i++) bulk.insert({}) bulk.execute(); print(i) s = db.c.stats(1024*1024) if ($t==1) print(s.size, s.totalIndexSize, s.size+s.totalIndexSize) if (s.size+s.totalIndexSize >= 8000) break } " done Start mongostat to monitor the cache statistics while paging in the b-trees using this command: mongo test --eval 'printjson(db.c.find({x:1}).hint({_id:1}).toArray())' While this is running you should observe the cache utilization climb to its limit, 80%. Now wait a minute or two (so that the b-trees haven't been accessed in a while), then trigger eviction of the test.c collection and _id index by creating a new test.d collection (20 MB will do): for t in $(seq 16); do mongo test --eval " every = 10000 for (var i=0; ; ) { var bulk = db.d.initializeUnorderedBulkOp(); for (var j=0; j<every; j++, i++) bulk.insert({}) bulk.execute(); print(i) s = db.d.stats(1024*1024) if ($t==1) print(s.size, s.totalIndexSize, s.size+s.totalIndexSize) if (s.size+s.totalIndexSize >= 20) break } " done Soon after this (within a minute) the sweep server will begin evicting the test.c collection and _id index b-trees; you will see cache utilization drop from 80% to 0% over the course of some time (a couple minutes, somewhat longer on Windows than Linux). During this time accesses to test.c block.

Top User Comments

xgen-internal-githook commented on Mon, 27 Apr 2015 07:17:33 +0000: Author: {u'username': u'michaelcahill', u'name': u'Michael Cahill', u'email': u'michael.cahill@mongodb.com'} Message: Discard trees from cache in the background. We used to keep handles locked while freeing their pages from cache (either for drops or when sweeping old handles). If an application thread attempted to open a cursor during one of these operations, it was forced to wait until the discard completed. With this change, handles are marked "dead", and readers will no longer use them. The sweep server will later discard dead trees from cache in the background, without holding any locks that application threads should block on. refs SERVER-17907, SERVER-18014 Branch: develop https://github.com/wiredtiger/wiredtiger/commit/440cbc76902432eb233b8a8bda1df1265bdd6e46 xgen-internal-githook commented on Mon, 13 Apr 2015 11:40:30 +0000: Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907. Branch: cursor-pin https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854 xgen-internal-githook commented on Fri, 10 Apr 2015 00:57:17 +0000: Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907. Branch: develop https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854 xgen-internal-githook commented on Wed, 8 Apr 2015 12:10:43 +0000: Author: {u'username': u'keithbostic', u'name': u'Keith Bostic', u'email': u'keith@wiredtiger.com'} Message: Don't sweep unless there's a reason we need to close file handles. Add a new configuration option, file_manager.open_handles that sets a minimum number of file handles that must be open before sweep runs, default is 250. Reference #1856, SERVER-17907. Branch: sweep-file-handle-minimum https://github.com/wiredtiger/wiredtiger/commit/709cc8d7ac85d31aeae2387b192092910e6cf854

Steps to Reproduce