Info

Per dupuisla, we need to reduce the number of file handles that we used to sort large amounts of data to avoid file handle exhaustion on various platforms. See SERVER-14572: I spent some time on the sorter.cpp and it is clear that you need to review this code. The FileIterator consume far too much file descriptors. There is no upper limit and increase the number of handle is just a poor fix. Why not merge all this in one file and use "seek" to move between the different block instead of creating thousand of files? In my case, I have something like 2000 of these temp files created, so 2048 will be on the low side.

Top User Comments

xgen-internal-githook commented on Tue, 11 Jun 2019 19:04:00 +0000: Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'} Message: SERVER-17010 each Sorter instance spills to a single file rather than a new file per spill to disk SERVER-38764 External sorter should use 64-bit integers for file offsets (cherry picked from commit 48d999c08304b6ede2a9d1f9d9db974b59fe97e2) Branch: v3.4 https://github.com/mongodb/mongo/commit/c1c761dd865308a15ff75748bf111d0a3ce366d6 xgen-internal-githook commented on Tue, 12 Mar 2019 00:47:27 +0000: Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'} Message: SERVER-17010 each Sorter instance spills to a single file rather than a new file per spill to disk (cherry picked from commit 2be7f2677a40a863f336d2964f456c9d87ddc838) SERVER-38764 External sorter should use 64-bit integers for file offsets (cherry picked from commit 9dafb7a3e3bafa463ab5951189b670965995dada) Branch: v3.6 https://github.com/mongodb/mongo/commit/48d999c08304b6ede2a9d1f9d9db974b59fe97e2 xgen-internal-githook commented on Mon, 25 Feb 2019 20:14:16 +0000: Author: {'name': 'Eric Milkie', 'username': 'milkie', 'email': 'milkie@10gen.com'} Message: SERVER-17010 additional fixes (didn't get merged with cherry-pick) Branch: v4.0 https://github.com/mongodb/mongo/commit/f906e0b21da0c0fb365687e4e04723a5af8d5002 xgen-internal-githook commented on Mon, 25 Feb 2019 19:19:36 +0000: Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'} Message: SERVER-17010 each Sorter instance spills to a single file rather than a new file per spill to disk (cherry picked from commit 2be7f2677a40a863f336d2964f456c9d87ddc838) Branch: v4.0 https://github.com/mongodb/mongo/commit/9491e6fb0ec2b38dd758d8acfe0e0aa11aa7bad9 xgen-internal-githook commented on Wed, 31 Oct 2018 12:50:48 +0000: Author: {'name': 'Dianna Hohensee', 'email': 'dianna.hohensee@10gen.com', 'username': 'DiannaHohensee'} Message: SERVER-17010 each Sorter instance spills to a single file rather than a new file per spill to disk Branch: master https://github.com/mongodb/mongo/commit/2be7f2677a40a863f336d2964f456c9d87ddc838 mars911 commented on Mon, 23 Oct 2017 06:44:06 +0000: How to solve this problem? redbeard0531 commented on Fri, 14 Jul 2017 14:43:14 +0000: It should work and be fairly easy to do. It is just a matter of changing the SortedFileWriter and sorter::FileIterator to work over ranges within a single file rather than having their own files, and having the few direct users of SortedFileWriter pass in the common file handle they want to use. ian@10gen.com commented on Fri, 14 Jul 2017 14:18:42 +0000: redbeard0531 What are your thoughts on how to approach this/complexity to do so? alexander.gorrod commented on Thu, 13 Jul 2017 06:23:30 +0000: This has been encountered by at least one customer recently - I've moved it into needs triage state, to ensure trigger a review of it's priority, design a path forward and schedule the work.

Steps to Reproduce