Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
We recently had this crash on v3.2.4 using zlib compression. Environment specs & stacks below. — ENV 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] MongoDB starting : pid=44128 port=27017 dbpath=/var/lib/mongodb 64-bit host=war-datamine7 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] db version v3.2.4 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] git version: e2ee9ffcf9f5a94fad76802e28cc978718bb7a30 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1f 6 Jan 2014 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] allocator: tcmalloc 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] modules: none 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] build environment: 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] distmod: ubuntu1404 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] distarch: x86_64 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] target_arch: x86_64 2016-07-14T13:03:38.486-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", replication: { replSetName: "rsDatalake" }, storage: { dbPath: "/var/lib/mongodb", directoryPerDB: true, engine: "wiredTiger", journal: { enabled: true }, wiredTiger: { collectionConfig: { blockCompressor: "zlib" }, engineConfig: { directoryForIndexes: true } } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log" } } ------- Stack Trace 28558 2016-07-14T06:21:18.784-0400 E STORAGE [repl writer worker 5] WiredTiger (0) [1468491678:783837][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: read checksum error for 4096B block at offset 17095036928: block header checksum of 3484947539 doesn't match expected checksum of 1241204689 2016-07-14T06:21:18.784-0400 E STORAGE [repl writer worker 5] WiredTiger (0) [1468491678:784337][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: db_logs/index/11--1216780978238540517.wt: encountered an illegal file format or internal value 2016-07-14T06:21:18.784-0400 E STORAGE [repl writer worker 5] WiredTiger (-31804) [1468491678:784353][39838:0x7f925ac42700], file:db_logs/index/11--1216780978238540517.wt, WT_CURSOR.insert: the process must exit and restart: WT_PANIC: WiredTiger library panic 2016-07-14T06:21:18.784-0400 I - [repl writer worker 5] Fatal Assertion 28558 2016-07-14T06:21:18.784-0400 I - [repl writer worker 5] ***aborting after fassert() failure 2016-07-14T06:21:18.819-0400 F - [repl writer worker 5] Got signal: 6 (Aborted). 0x12f3502 0x12f2659 0x12f2e62 0x7f926ce67330 0x7f926cac8c37 0x7f926cacc028 0x127d9d2 0x1078ef3 0x1a378ec 0x1a37a8d 0x1a37e74 0x1971b16 0x198e7fa 0x1993d90 0x19b3be8 0x198229f 0x19cf28e 0x105696a 0x105a027 0xc91a0f 0xae4fc9 0xae936f 0xae9410 0xac93f5 0xac9761 0xac990b 0xe99c51 0xf2b58e 0xf253e0 0xf27026 0xf27c26 0xf2aedb 0x1288051 0x12889b9 0x1289510 0x7f926d642a60 0x7f926ce5f184 0x7f926cb8c37d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"EF3502","s":"_ZN5mongo15printStackTraceERSo"},{"b":"400000","o":"EF2659"},{"b":"400000","o":"EF2E62"},{"b":"7F926CE57000","o":"10330"},{"b":"7F926CA92000","o":"36C37","s":"gsignal"},{"b":"7F926CA92000","o":"3A028","s":"abort"},{"b":"400000","o":"E7D9D2","s":"_ZN5mongo13fassertFailedEi"},{"b":"400000","o":"C78EF3"},{"b":"400000","o":"16378EC","s":"__wt_eventv"},{"b":"400000","o":"1637A8D","s":"__wt_err"},{"b":"400000","o":"1637E74","s":"__wt_panic"},{"b":"400000","o":"1571B16","s":"__wt_bm_read"},{"b":"400000","o":"158E7FA","s":"__wt_bt_read"},{"b":"400000","o":"1593D90","s":"__wt_page_in_func"},{"b":"400000","o":"15B3BE8","s":"__wt_row_search"},{"b":"400000","o":"158229F","s":"__wt_btcur_insert"},{"b":"400000","o":"15CF28E"},{"b":"400000","o":"C5696A","s":"_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"C5A027","s":"_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb"},{"b":"400000","o":"891A0F","s":"_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl"},{"b":"400000","o":"6E4FC9","s":"_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},{"b":"400000","o":"6E936F","s":"_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE"},{"b":"400000","o":"6E9410","s":"_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE"},{"b":"400000","o":"6C93F5","s":"_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b"},{"b":"400000","o":"6C9761","s":"_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb"},{"b":"400000","o":"6C990B","s":"_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb"},{"b":"400000","o":"A99C51","s":"_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb"},{"b":"400000","o":"B2B58E","s":"_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b"},{"b":"400000","o":"B253E0"},{"b":"400000","o":"B27026","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE"},{"b":"400000","o":"B27C26","s":"_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb"},{"b":"400000","o":"B2AEDB","s":"_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE"},{"b":"400000","o":"E88051","s":"_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE"},{"b":"400000","o":"E889B9","s":"_ZN5mongo10ThreadPool13_consumeTasksEv"},{"b":"400000","o":"E89510","s":"_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs"},{"b":"7F926D591000","o":"B1A60"},{"b":"7F926CE57000","o":"8184"},{"b":"7F926CA92000","o":"FA37D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.2.4", "gitVersion" : "e2ee9ffcf9f5a94fad76802e28cc978718bb7a30", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.2.0-34-generic", "version" : "#39~14.04.1-Ubuntu SMP Fri Mar 11 11:38:02 UTC 2016", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "EF46210F8976780D45B811C3540FECB9E734EABE" }, { "b" : "7FFDFC9A0000", "elfType" : 3, "buildId" : "89FBCFEB3FFCA6A9AAB57384D5EC6D1C8AB68009" }, { "b" : "7F926E07D000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "74864DB9D5F69D39A67E4755012FB6573C469B3D" }, { "b" : "7F926DCA1000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "AAE7CFF8351B730830BDBCE0DCABBE06574B7144" }, { "b" : "7F926DA99000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "E2A6DD5048A0A051FD61043BDB69D8CC68192AB7" }, { "b" : "7F926D895000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "DA9B8C234D0FE9FD8CAAC8970A7EC1B6C8F6623F" }, { "b" : "7F926D591000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "76190E922AF7457D078F75C9B15FA184E83EB506" }, { "b" : "7F926D28B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "D144258E614900B255A31F3FD2283A878670D5BC" }, { "b" : "7F926D075000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "36311B4457710AE5578C4BF00791DED7359DBB92" }, { "b" : "7F926CE57000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "31E9F21AE8C10396171F1E13DA15780986FA696C" }, { "b" : "7F926CA92000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "CF699A15CAAE64F50311FC4655B86DC39A479789" }, { "b" : "7F926E2DC000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "D0F537904076D73F29E4A37341F8A449E2EF6CD0" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f3502] mongod(+0xEF2659) [0x12f2659] mongod(+0xEF2E62) [0x12f2e62] libpthread.so.0(+0x10330) [0x7f926ce67330] libc.so.6(gsignal+0x37) [0x7f926cac8c37] libc.so.6(abort+0x148) [0x7f926cacc028] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127d9d2] mongod(+0xC78EF3) [0x1078ef3] mongod(__wt_eventv+0x40C) [0x1a378ec] mongod(__wt_err+0x8D) [0x1a37a8d] mongod(__wt_panic+0x24) [0x1a37e74] mongod(__wt_bm_read+0x76) [0x1971b16] mongod(__wt_bt_read+0x1EA) [0x198e7fa] mongod(__wt_page_in_func+0x180) [0x1993d90] mongod(__wt_row_search+0x698) [0x19b3be8] mongod(__wt_btcur_insert+0x45F) [0x198229f] mongod(+0x15CF28E) [0x19cf28e] mongod(_ZN5mongo23WiredTigerIndexStandard7_insertEP11__wt_cursorRKNS_7BSONObjERKNS_8RecordIdEb+0x12A) [0x105696a] mongod(_ZN5mongo15WiredTigerIndex6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdEb+0xC7) [0x105a027] mongod(_ZN5mongo17IndexAccessMethod6insertEPNS_16OperationContextERKNS_7BSONObjERKNS_8RecordIdERKNS_19InsertDeleteOptionsEPl+0x19F) [0xc91a0f] mongod(_ZN5mongo12IndexCatalog21_indexFilteredRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x109) [0xae4fc9] mongod(_ZN5mongo12IndexCatalog13_indexRecordsEPNS_16OperationContextEPNS_17IndexCatalogEntryERKSt6vectorINS_10BsonRecordESaIS6_EE+0x11F) [0xae936f] mongod(_ZN5mongo12IndexCatalog12indexRecordsEPNS_16OperationContextERKSt6vectorINS_10BsonRecordESaIS4_EE+0x80) [0xae9410] mongod(_ZN5mongo10Collection16_insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_b+0x325) [0xac93f5] mongod(_ZN5mongo10Collection15insertDocumentsEPNS_16OperationContextEN9__gnu_cxx17__normal_iteratorIPKNS_7BSONObjESt6vectorIS5_SaIS5_EEEESB_bb+0x1B1) [0xac9761] mongod(_ZN5mongo10Collection14insertDocumentEPNS_16OperationContextERKNS_7BSONObjEbb+0x6B) [0xac990b] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0xF41) [0xe99c51] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf2b58e] mongod(+0xB253E0) [0xf253e0] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0x mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf27c26] mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf2aedb] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x1288051] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12889b9] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x1289510] libstdc++.so.6(+0xB1A60) [0x7f926d642a60] libpthread.so.0(+0x8184) [0x7f926ce5f184] libc.so.6(clone+0x6D) [0x7f926cb8c37d] ----- END BACKTRACE ----- Booting up server after crash produces ------- Stack Trace 28559 2016-07-14T13:03:53.379-0400 E STORAGE [repl writer worker 8] WiredTiger (0) [1468515833:379294][44128:0x7ff178aa1700] 2016-07-14T13:03:53.379-0400 E STORAGE [repl writer worker 8] WiredTiger (0) [1468515833:379422][44128:0x7ff178aa1700] 2016-07-14T13:03:53.379-0400 E STORAGE [repl writer worker 8] WiredTiger (-31804) [1468515833:379455][44128:0x7ff178aa1700] 2016-07-14T13:03:53.379-0400 I - [repl writer worker 4] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 13] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 9] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 2] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 9] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 2] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 8] Fatal Assertion 28558 2016-07-14T13:03:53.379-0400 I - [repl writer worker 11] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 12] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 3] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 3] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 13] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 14] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 14] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [rsSync] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [rsSync] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 11] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 4] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 12] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 6] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [repl writer worker 6] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 0] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [repl writer worker 0] ***aborting after fassert() failure 2016-07-14T13:03:53.379-0400 I - [repl writer worker 7] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 8] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [repl writer worker 7] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [repl writer worker 5] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [repl writer worker 15] Fatal Assertion 28559 2016-07-14T13:03:53.379-0400 I - [repl writer worker 10] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [repl writer worker 15] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [repl writer worker 10] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [repl writer worker 1] Fatal Assertion 28559 2016-07-14T13:03:53.380-0400 I - [repl writer worker 1] ***aborting after fassert() failure 2016-07-14T13:03:53.380-0400 I - [repl writer worker 5] ***aborting after fassert() failure 2016-07-14T13:03:53.397-0400 F - [repl writer worker 9] Got signal: 6 (Aborted). 0x12f3502 0x12f2659 0x12f2e62 0x7ff18c4c9330 0x7ff18c12ac37 0x7ff18c12e028 0x127d9d2 0x1079345 0x106e3a7 0xc13089 0xbdaffa 0xc0ff04 0xe0ac85 0xe0b349 0xe0b445 0xd1d479 0xe9a1dc 0xf2b58e 0xf253e0 0xf27026 0xf27c26 0xf2aedb 0x1288051 0x12889b9 0x1289510 0x7ff18cca4a60 0x7ff18c4c1184 0x7ff18c1ee37d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000" mongod(_ZN5mongo15printStackTraceERSo+0x32) [0x12f3502] mongod(+0xEF2659) [0x12f2659] mongod(+0xEF2E62) [0x12f2e62] libpthread.so.0(+0x10330) [0x7ff18c4c9330] libc.so.6(gsignal+0x37) [0x7ff18c12ac37] libc.so.6(abort+0x148) [0x7ff18c12e028] mongod(_ZN5mongo13fassertFailedEi+0x82) [0x127d9d2] mongod(_ZN5mongo17wtRCToStatus_slowEiPKc+0x355) [0x1079345] mongod(_ZN5mongo21WiredTigerRecordStore6Cursor9seekExactERKNS_8RecordIdE+0x157) [0x106e3a7] mongod(_ZN5mongo16WorkingSetCommon5fetchEPNS_16OperationContextEPNS_10WorkingSetEmNS_11unowned_ptrINS_20SeekableRecordCursorEEE+0x99) [0xc13089] mongod(_ZN5mongo11IDHackStage4workEPm+0x21A) [0xbdaffa] mongod(_ZN5mongo11UpdateStage4workEPm+0x394) [0xc0ff04] mongod(_ZN5mongo12PlanExecutor11getNextImplEPNS_11SnapshottedINS_7BSONObjEEEPNS_8RecordIdE+0x275) [0xe0ac85] mongod(_ZN5mongo12PlanExecutor7getNextEPNS_7BSONObjEPNS_8RecordIdE+0x39) [0xe0b349] mongod(_ZN5mongo12PlanExecutor11executePlanEv+0x55) [0xe0b445] mongod(_ZN5mongo6updateEPNS_16OperationContextEPNS_8DatabaseERKNS_13UpdateRequestEPNS_7OpDebugE+0x179) [0xd1d479] mongod(_ZN5mongo4repl21applyOperation_inlockEPNS_16OperationContextEPNS_8DatabaseERKNS_7BSONObjEb+0x14CC) [0xe9a1dc] mongod(_ZNSt17_Function_handlerIFN5mongo6StatusEPNS0_16OperationContextEPNS0_8DatabaseERKNS0_7BSONObjEbEPS9_E9_M_invokeERKSt9_Any_dataS3_S5_S8_b+0x1E) [0xf2b58e] mongod(+0xB253E0) [0xf253e0] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEbSt8functionIFNS_6StatusES3_PNS_8DatabaseES6_bEES7_IFS8_S3_S6_EES7_IFvvEE+0x336) [0xf27026] mongod(_ZN5mongo4repl8SyncTail9syncApplyEPNS_16OperationContextERKNS_7BSONObjEb+0xE6) [0xf27c26] mongod(_ZN5mongo4repl14multiSyncApplyERKSt6vectorINS_7BSONObjESaIS2_EEPNS0_8SyncTailE+0x9B) [0xf2aedb] mongod(_ZN5mongo10ThreadPool10_doOneTaskEPSt11unique_lockISt5mutexE+0x121) [0x1288051] mongod(_ZN5mongo10ThreadPool13_consumeTasksEv+0xA9) [0x12889b9] mongod(_ZN5mongo10ThreadPool17_workerThreadBodyEPS0_RKSs+0x100) [0x1289510] libstdc++.so.6(+0xB1A60) [0x7ff18cca4a60] libpthread.so.0(+0x8184) [0x7ff18c4c1184] libc.so.6(clone+0x6D) [0x7ff18c1ee37d] ----- END BACKTRACE -----
thomas.schubert commented on Mon, 25 Jul 2016 20:48:29 +0000: Hi GACTON, Thank you for answering my questions. Unfortunately, I do not see anything in your response that clearly points to a root cause either in your disk system, MongoDB, or system configuration settings. Since we are unable to reproduce this issue, I am closing this ticket. Please open a new ticket if this issue occurs again. Kind regards, Thomas gacton commented on Thu, 14 Jul 2016 20:28:17 +0000: 1. No other errors. A number of 'getmore' operations based on running queries. 2. Locally attached device - Massive 8TB HDD. Redundancy achieved via cluster replication. Volume management: /dev/sdb1 /var/lib/mongodb ext4 rw,errors=remount-ro 0 0 3. No. 4. No 5. No need for backups. The data added into this cluster is from backup archives. Thanks! thomas.schubert commented on Thu, 14 Jul 2016 18:16:28 +0000: Hi GACTON, These error messages indicate some data files have become corrupt in some way. I've assembled a list of routine questions about data storage and the configuration of your environment, which will help us examine this issue. However, please note that it may be challenging to identify the root cause of the the corruption without a straightforward reproduction. Around this operation, were there any other server errors logged? What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using? Have you manipulated (copied or moved) the underlying database files? If so, was mongod running? Have you ever restored this instance from backups? What method do you use to create backups? I would recommend a clean resync from a node that is not affected. Thank you, Thomas