Info

Hi There, We have got snapshot mongodb directory from broken server hardware, when we are trying to start mongo 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] MongoDB starting : pid=19431 port=27017 dbpath=/var/lib/mongodb 64-bit host=lang 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] db version v3.6.1 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] allocator: tcmalloc 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] modules: none 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] build environment: 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] distarch: x86_64 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] target_arch: x86_64 2018-01-18T23:32:02.895+0300 I CONTROL [initandlisten] options: { config: "/etc/mongodb.conf", storage: { dbPath: "/var/lib/mongodb" }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", quiet: true } } 2018-01-18T23:32:02.895+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2018-01-18T23:32:02.895+0300 I - [initandlisten] Detected data files in /var/lib/mongodb created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-01-18T23:32:02.895+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-01-18T23:32:02.896+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2018-01-18T23:32:02.977+0300 I STORAGE [initandlisten] WiredTiger message [1516307522:977147][19431:0x7f0765f38a80], txn-recover: Main recovery loop: starting at 59/128 2018-01-18T23:32:02.977+0300 I STORAGE [initandlisten] WiredTiger message [1516307522:977419][19431:0x7f0765f38a80], txn-recover: Recovering log 59 through 60 2018-01-18T23:32:02.984+0300 I STORAGE [initandlisten] WiredTiger message [1516307522:984322][19431:0x7f0765f38a80], txn-recover: Recovering log 60 through 60 2018-01-18T23:32:03.062+0300 E STORAGE [initandlisten] WiredTiger error (0) [1516307523:62572][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: read checksum error for 4096B block at offset 28672: block header checksum of 0 doesn't match expected checksum of 1074498338 2018-01-18T23:32:03.062+0300 E STORAGE [initandlisten] WiredTiger error (0) [1516307523:62643][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: _mdb_catalog.wt: encountered an illegal file format or internal value: (__wt_block_read_off, 227) 2018-01-18T23:32:03.062+0300 E STORAGE [initandlisten] WiredTiger error (-31804) [1516307523:62669][19431:0x7f0765f38a80], file:_mdb_catalog.wt, WT_SESSION.open_cursor: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-01-18T23:32:03.062+0300 F - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2018-01-18T23:32:03.062+0300 F - [initandlisten] ***aborting after fassert() failure 2018-01-18T23:32:03.090+0300 F - [initandlisten] Got signal: 6 (Aborted). 0x557b6eaea8da 0x557b6eae9b9e 0x557b6eaea072 0x7f076325bdb0 0x7f0762ec7860 0x7f0762ec8ec9 0x557b6d41c8f7 0x557b6d4ed876 0x7f0765a0bc0e 0x7f0765a0be23 0x7f0765a0c177 0x7f0765a139d3 0x7f0765a13f23 0x7f0765a1035e 0x7f0765a15212 0x7f0765a31825 0x7f0765a774a9 0x7f0765b052ad 0x7f0765b057ef 0x7f0765b05aa0 0x7f0765a86ceb 0x7f0765b01d79 0x7f0765a9f5fe 0x7f0765b01c69 0x7f0765b021fa 0x557b6d4e71b6 0x557b6d4e6a4c 0x557b6d4dc588 0x557b6d4dc636 0x557b6d4e1ee1 0x557b6d4cce98 0x557b6d502160 0x557b6d4b7397 0x557b6d592bf3 0x557b6d494011 0x557b6d4975fa 0x557b6d41e449 0x7f0762eb3f4a 0x557b6d48141a ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"557B6CBC6000","o":"1F248DA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"557B6CBC6000","o":"1F23B9E"},{"b":"557B6CBC6000","o":"1F24072"},{"b":"7F076324A000","o":"11DB0"},{"b":"7F0762E93000","o":"34860","s":"gsignal"},{"b":"7F0762E93000","o":"35EC9","s":"abort"},{"b":"557B6CBC6000","o":"8568F7","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"557B6CBC6000","o":"927876"},{"b":"7F07659EC000","o":"1FC0E","s":"__wt_eventv"},{"b":"7F07659EC000","o":"1FE23","s":"__wt_err"},{"b":"7F07659EC000","o":"20177","s":"__wt_panic"},{"b":"7F07659EC000","o":"279D3","s":"__wt_block_extlist_read"},{"b":"7F07659EC000","o":"27F23","s":"__wt_block_extlist_read_avail"},{"b":"7F07659EC000","o":"2435E","s":"__wt_block_checkpoint_load"},{"b":"7F07659EC000","o":"29212"},{"b":"7F07659EC000","o":"45825","s":"__wt_btree_open"},{"b":"7F07659EC000","o":"8B4A9","s":"__wt_conn_dhandle_open"},{"b":"7F07659EC000","o":"1192AD","s":"__wt_session_get_dhandle"},{"b":"7F07659EC000","o":"1197EF","s":"__wt_session_get_dhandle"},{"b":"7F07659EC000","o":"119AA0","s":"__wt_session_get_btree_ckpt"},{"b":"7F07659EC000","o":"9ACEB","s":"__wt_curfile_open"},{"b":"7F07659EC000","o":"115D79"},{"b":"7F07659EC000","o":"B35FE","s":"__wt_curtable_open"},{"b":"7F07659EC000","o":"115C69"},{"b":"7F07659EC000","o":"1161FA"},{"b":"557B6CBC6000","o":"9211B6","s":"_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb"},{"b":"557B6CBC6000","o":"920A4C","s":"_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE"},{"b":"557B6CBC6000","o":"916588","s":"_ZN5mongo31WiredTigerRecordStoreCursorBaseC2EPNS_16OperationContextERKNS_21WiredTigerRecordStoreEb"},{"b":"557B6CBC6000","o":"916636","s":"_ZNK5mongo29StandardWiredTigerRecordStore9getCursorEPNS_16OperationContextEb"},{"b":"557B6CBC6000","o":"91BEE1","s":"_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE"},{"b":"557B6CBC6000","o":"906E98","s":"_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE"},{"b":"557B6CBC6000","o":"93C160","s":"_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE"},{"b":"557B6CBC6000","o":"8F1397"},{"b":"557B6CBC6000","o":"9CCBF3","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"557B6CBC6000","o":"8CE011"},{"b":"557B6CBC6000","o":"8D15FA","s":"_ZN5mongo11mongoDbMainEiPPcS1_"},{"b":"557B6CBC6000","o":"858449","s":"main"},{"b":"7F0762E93000","o":"20F4A","s":"__libc_start_main"},{"b":"557B6CBC6000","o":"8BB41A","s":"_start"}],"processInfo":{ "mongodbVersion" : "3.6.1", "gitVersion" : "025d4f4fe61efd1fb6f0005be20cb45a004093d1", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.14.13-1-ARCH", "version" : "#1 SMP PREEMPT Wed Jan 10 11:14:50 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "557B6CBC6000", "elfType" : 3, "buildId" : "029D000C7CD46EBE64E61D75949D0CCA9AFE8548" }, { "b" : "7FFCB79F2000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "3832AB6026917A8D56323E7017DC052A5FC0F5C6" }, { "b" : "7F07659EC000", "path" : "/usr/lib/libwiredtiger-3.0.0.so", "elfType" : 3, "buildId" : "AF35154F05948BA92AEFEC42B9B37264437FAA8E" }, { "b" : "7F07657CF000", "path" : "/usr/lib/libsasl2.so.3", "elfType" : 3, "buildId" : "FD2BC2C7EB41B83A66FF6D099037532027A47916" }, { "b" : "7F0765574000", "path" : "/usr/lib/libstemmer.so.0", "elfType" : 3, "buildId" : "AD50FFCB248B0279A46FA04967455B1EFD942AF0" }, { "b" : "7F076535D000", "path" : "/usr/lib/libz.so.1", "elfType" : 3, "buildId" : "9D65B31D958D72054843331A9D549B1D12ADA045" }, { "b" : "7F0765155000", "path" : "/usr/lib/libsnappy.so.1", "elfType" : 3, "buildId" : "4945D214906C5C184B044DF00ED23A3A4DC08166" }, { "b" : "7F0764ED8000", "path" : "/usr/lib/libyaml-cpp.so.0.5", "elfType" : 3, "buildId" : "4CA43F102CA009469006E0FA5A595D110E72A88A" }, { "b" : "7F0764C65000", "path" : "/usr/lib/libpcre.so.1", "elfType" : 3, "buildId" : "00A48F31C879241B1621FA0EE8A77E24333D006B" }, { "b" : "7F0764A5C000", "path" : "/usr/lib/libpcrecpp.so.0", "elfType" : 3, "buildId" : "A97FFE1C498EB6B2E383ABADF091EAF02D7FCA18" }, { "b" : "7F0764845000", "path" : "/usr/lib/libresolv.so.2", "elfType" : 3, "buildId" : "C37EEAC29624BF0F52FA85E0C5035B0ABAEDC100" }, { "b" : "7F07645DB000", "path" : "/usr/lib/libssl.so.1.1", "elfType" : 3, "buildId" : "8B3582F677DE36D14D30BECE4A92F1CD19026299" }, { "b" : "7F076415E000", "path" : "/usr/lib/libcrypto.so.1.1", "elfType" : 3, "buildId" : "05A9A6C52AACA0043A1D4A046193015D972AC092" }, { "b" : "7F0763F56000", "path" : "/usr/lib/librt.so.1", "elfType" : 3, "buildId" : "D520E36F95020BC09653CBFA3E3533E2F823713C" }, { "b" : "7F0763D52000", "path" : "/usr/lib/libdl.so.2", "elfType" : 3, "buildId" : "F4C88B82DD709ACA91EB652FFDBD91900B8A908D" }, { "b" : "7F07639CB000", "path" : "/usr/lib/libstdc++.so.6", "elfType" : 3, "buildId" : "14EC457A4EF13B537BA15A26301F1512CE3277F5" }, { "b" : "7F076367F000", "path" : "/usr/lib/libm.so.6", "elfType" : 3, "buildId" : "FA35622098EC404C4787159F0AB09FB2F6C5466F" }, { "b" : "7F0763468000", "path" : "/usr/lib/libgcc_s.so.1", "elfType" : 3, "buildId" : "BC76DEC7E4673A3AF72A6B26BBB7916359B9802F" }, { "b" : "7F076324A000", "path" : "/usr/lib/libpthread.so.0", "elfType" : 3, "buildId" : "F2D89412406AED9373964B49A96BA88BAA53A0BC" }, { "b" : "7F0762E93000", "path" : "/usr/lib/libc.so.6", "elfType" : 3, "buildId" : "48CE2E5F2FF41F8C7911E882A033C8C222B7F209" }, { "b" : "7F0762A9E000", "path" : "/usr/lib/libtcmalloc.so.4", "elfType" : 3, "buildId" : "475D699EF3D447B31EBBC2ED5B7EEE7702C85447" }, { "b" : "7F0762889000", "path" : "/usr/lib/liblz4.so.1", "elfType" : 3, "buildId" : "A274C0E4DBC85A4C859DDFFE638FBB9AE9A156AC" }, { "b" : "7F0765D61000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "E372512D034C6ABD6C233F9CCE96597653F372D3" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x557b6eaea8da] mongod(+0x1F23B9E) [0x557b6eae9b9e] mongod(+0x1F24072) [0x557b6eaea072] libpthread.so.0(+0x11DB0) [0x7f076325bdb0] libc.so.6(gsignal+0x110) [0x7f0762ec7860] libc.so.6(abort+0x1C9) [0x7f0762ec8ec9] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x557b6d41c8f7] mongod(+0x927876) [0x557b6d4ed876] libwiredtiger-3.0.0.so(__wt_eventv+0x36D) [0x7f0765a0bc0e] libwiredtiger-3.0.0.so(__wt_err+0xA3) [0x7f0765a0be23] libwiredtiger-3.0.0.so(__wt_panic+0x35) [0x7f0765a0c177] libwiredtiger-3.0.0.so(__wt_block_extlist_read+0x93) [0x7f0765a139d3] libwiredtiger-3.0.0.so(__wt_block_extlist_read_avail+0x23) [0x7f0765a13f23] libwiredtiger-3.0.0.so(__wt_block_checkpoint_load+0x2CE) [0x7f0765a1035e] libwiredtiger-3.0.0.so(+0x29212) [0x7f0765a15212] libwiredtiger-3.0.0.so(__wt_btree_open+0xB15) [0x7f0765a31825] libwiredtiger-3.0.0.so(__wt_conn_dhandle_open+0x229) [0x7f0765a774a9] libwiredtiger-3.0.0.so(__wt_session_get_dhandle+0xDD) [0x7f0765b052ad] libwiredtiger-3.0.0.so(__wt_session_get_dhandle+0x61F) [0x7f0765b057ef] libwiredtiger-3.0.0.so(__wt_session_get_btree_ckpt+0x150) [0x7f0765b05aa0] libwiredtiger-3.0.0.so(__wt_curfile_open+0x4B) [0x7f0765a86ceb] libwiredtiger-3.0.0.so(+0x115D79) [0x7f0765b01d79] libwiredtiger-3.0.0.so(__wt_curtable_open+0xCE) [0x7f0765a9f5fe] libwiredtiger-3.0.0.so(+0x115C69) [0x7f0765b01c69] libwiredtiger-3.0.0.so(+0x1161FA) [0x7f0765b021fa] mongod(_ZN5mongo17WiredTigerSession9getCursorERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmb+0x76) [0x557b6d4e71b6] mongod(_ZN5mongo16WiredTigerCursorC1ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEmbPNS_16OperationContextE+0x4C) [0x557b6d4e6a4c] mongod(_ZN5mongo31WiredTigerRecordStoreCursorBaseC2EPNS_16OperationContextERKNS_21WiredTigerRecordStoreEb+0x58) [0x557b6d4dc588] mongod(_ZNK5mongo29StandardWiredTigerRecordStore9getCursorEPNS_16OperationContextEb+0x86) [0x557b6d4dc636] mongod(_ZN5mongo21WiredTigerRecordStore19postConstructorInitEPNS_16OperationContextE+0x41) [0x557b6d4e1ee1] mongod(_ZN5mongo18WiredTigerKVEngine21getGroupedRecordStoreEPNS_16OperationContextENS_10StringDataES3_RKNS_17CollectionOptionsENS_8KVPrefixE+0x2D8) [0x557b6d4cce98] mongod(_ZN5mongo15KVStorageEngineC1EPNS_8KVEngineERKNS_22KVStorageEngineOptionsESt8functionIFSt10unique_ptrINS_26KVDatabaseCatalogEntryBaseESt14default_deleteIS8_EENS_10StringDataEPS0_EE+0x11D0) [0x557b6d502160] mongod(+0x8F1397) [0x557b6d4b7397] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x613) [0x557b6d592bf3] mongod(+0x8CE011) [0x557b6d494011] mongod(_ZN5mongo11mongoDbMainEiPPcS1_+0x61A) [0x557b6d4975fa] mongod(main+0x9) [0x557b6d41e449] libc.so.6(__libc_start_main+0xEA) [0x7f0762eb3f4a] mongod(_start+0x2A) [0x557b6d48141a] ----- END BACKTRACE ----- Then repair attempt deleting all our data mongod --repair --dbpath /var/lib/mongodb/ 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] MongoDB starting : pid=19599 port=27017 dbpath=/var/lib/mongodb/ 64-bit host=lang 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] db version v3.6.1 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] git version: 025d4f4fe61efd1fb6f0005be20cb45a004093d1 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.1.0g 2 Nov 2017 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] allocator: tcmalloc 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] modules: none 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] build environment: 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] distarch: x86_64 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] target_arch: x86_64 2018-01-18T23:33:42.861+0300 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: "/var/lib/mongodb/" } } 2018-01-18T23:33:42.861+0300 W - [initandlisten] Detected unclean shutdown - /var/lib/mongodb/mongod.lock is not empty. 2018-01-18T23:33:42.862+0300 I - [initandlisten] Detected data files in /var/lib/mongodb/ created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2018-01-18T23:33:42.862+0300 W STORAGE [initandlisten] Recovering data from the last clean checkpoint. 2018-01-18T23:33:42.862+0300 I STORAGE [initandlisten] Detected WT journal files. Running recovery from last checkpoint. 2018-01-18T23:33:42.862+0300 I STORAGE [initandlisten] journal to nojournal transition config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress), 2018-01-18T23:33:42.960+0300 I STORAGE [initandlisten] WiredTiger message [1516307622:960077][19599:0x7f54813f0a80], txn-recover: Main recovery loop: starting at 60/256 2018-01-18T23:33:42.960+0300 I STORAGE [initandlisten] WiredTiger message [1516307622:960231][19599:0x7f54813f0a80], txn-recover: Recovering log 60 through 61 2018-01-18T23:33:42.999+0300 I STORAGE [initandlisten] WiredTiger message [1516307622:999142][19599:0x7f54813f0a80], txn-recover: Recovering log 61 through 61 2018-01-18T23:33:43.153+0300 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=7477M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),statistics_log=(wait=0),verbose=(recovery_progress),,log=(enabled=false), 2018-01-18T23:33:43.205+0300 I STORAGE [initandlisten] Repairing size cache 2018-01-18T23:33:43.205+0300 I STORAGE [initandlisten] Verify succeeded on uri table:sizeStorer. Not salvaging. 2018-01-18T23:33:43.206+0300 I STORAGE [initandlisten] Repairing catalog metadata 2018-01-18T23:33:43.206+0300 E STORAGE [initandlisten] WiredTiger error (0) [1516307623:206574][19599:0x7f54813f0a80], file:_mdb_catalog.wt, WT_SESSION.verify: read checksum error for 4096B block at offset 28672: block header checksum of 0 doesn't match expected checksum of 1074498338 2018-01-18T23:33:43.206+0300 I STORAGE [initandlisten] Verify failed on uri table:_mdb_catalog. Running a salvage operation. 2018-01-18T23:33:43.214+0300 I CONTROL [initandlisten] 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** WARNING: You are running this process as the root user, which is not recommended. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** WARNING: This server is bound to localhost. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** Remote systems will be unable to connect to this server. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** Start the server with --bind_ip to specify which IP 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** addresses it should serve responses from, or with --bind_ip_all to 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** bind to all interfaces. If this behavior is desired, start the 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** server with --bind_ip 127.0.0.1 to disable this warning. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'. 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] ** We suggest setting it to 'never' 2018-01-18T23:33:43.215+0300 I CONTROL [initandlisten] 2018-01-18T23:33:43.215+0300 I STORAGE [initandlisten] createCollection: admin.system.version with provided UUID: a3471fa1-b9a7-4066-9c51-f9114d90f70e 2018-01-18T23:33:43.278+0300 I COMMAND [initandlisten] setting featureCompatibilityVersion to 3.6 2018-01-18T23:33:43.278+0300 I STORAGE [initandlisten] repairDatabase admin 2018-01-18T23:33:43.278+0300 I STORAGE [initandlisten] Repairing collection admin.system.version 2018-01-18T23:33:43.306+0300 I STORAGE [initandlisten] Verify succeeded on uri table:collection-0--803559634327283876. Not salvaging. 2018-01-18T23:33:43.352+0300 I INDEX [initandlisten] build index on: admin.system.version properties: { v: 2, key: { _id: 1 }, name: "_id_", ns: "admin.system.version" } 2018-01-18T23:33:43.352+0300 I INDEX [initandlisten] building index using bulk method; build may temporarily use up to 500 megabytes of RAM 2018-01-18T23:33:43.437+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-0--2170319704746865155 2018-01-18T23:33:43.455+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-0--2919941413266653091 2018-01-18T23:33:43.472+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-0--3119660283819234981 2018-01-18T23:33:43.489+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-0--5607842886892814276 2018-01-18T23:33:43.508+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-14--2919941413266653091 2018-01-18T23:33:43.539+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-16--2919941413266653091 2018-01-18T23:33:43.559+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-18--2919941413266653091 2018-01-18T23:33:43.585+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-2--2351367194744710572 2018-01-18T23:33:43.614+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-2--3119660283819234981 2018-01-18T23:33:43.639+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-2--5607842886892814276 2018-01-18T23:33:43.663+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-26--2919941413266653091 2018-01-18T23:33:43.682+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-30--2919941413266653091 2018-01-18T23:33:43.706+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-32--2919941413266653091 2018-01-18T23:33:43.726+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-34--2919941413266653091 2018-01-18T23:33:43.745+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-4--2351367194744710572 2018-01-18T23:33:43.764+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-4--5607842886892814276 2018-01-18T23:33:43.789+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-6--2351367194744710572 2018-01-18T23:33:43.808+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-6--5607842886892814276 2018-01-18T23:33:43.828+0300 I STORAGE [initandlisten] Dropping unknown ident: collection-8--5607842886892814276 2018-01-18T23:33:43.847+0300 I STORAGE [initandlisten] Dropping unknown ident: index-0-7687852694704450203 2018-01-18T23:33:43.869+0300 I STORAGE [initandlisten] Dropping unknown ident: index-1-7687852694704450203 2018-01-18T23:33:43.887+0300 I STORAGE [initandlisten] Dropping unknown ident: index-10-7687852694704450203 2018-01-18T23:33:43.911+0300 I STORAGE [initandlisten] Dropping unknown ident: index-11-7687852694704450203 2018-01-18T23:33:43.928+0300 I STORAGE [initandlisten] Dropping unknown ident: index-12-7687852694704450203 2018-01-18T23:33:43.954+0300 I STORAGE [initandlisten] Dropping unknown ident: index-13-7687852694704450203 2018-01-18T23:33:43.971+0300 I STORAGE [initandlisten] Dropping unknown ident: index-14-7687852694704450203 2018-01-18T23:33:43.997+0300 I STORAGE [initandlisten] Dropping unknown ident: index-15-7687852694704450203 2018-01-18T23:33:44.015+0300 I STORAGE [initandlisten] Dropping unknown ident: index-16-7687852694704450203 2018-01-18T23:33:44.033+0300 I STORAGE [initandlisten] Dropping unknown ident: index-17-7687852694704450203 2018-01-18T23:33:44.051+0300 I STORAGE [initandlisten] Dropping unknown ident: index-18-7687852694704450203 2018-01-18T23:33:44.073+0300 I STORAGE [initandlisten] Dropping unknown ident: index-19-7687852694704450203 2018-01-18T23:33:44.092+0300 I STORAGE [initandlisten] Dropping unknown ident: index-2-7687852694704450203 2018-01-18T23:33:44.119+0300 I STORAGE [initandlisten] Dropping unknown ident: index-20-7687852694704450203 2018-01-18T23:33:44.142+0300 I STORAGE [initandlisten] Dropping unknown ident: index-21-7687852694704450203 2018-01-18T23:33:44.164+0300 I STORAGE [initandlisten] Dropping unknown ident: index-22-7687852694704450203 2018-01-18T23:33:44.188+0300 I STORAGE [initandlisten] Dropping unknown ident: index-23-7687852694704450203 2018-01-18T23:33:44.207+0300 I STORAGE [initandlisten] Dropping unknown ident: index-24-7687852694704450203 2018-01-18T23:33:44.226+0300 I STORAGE [initandlisten] Dropping unknown ident: index-25-7687852694704450203 2018-01-18T23:33:44.244+0300 I STORAGE [initandlisten] Dropping unknown ident: index-3-7687852694704450203 2018-01-18T23:33:44.267+0300 I STORAGE [initandlisten] Dropping unknown ident: index-4-7687852694704450203 2018-01-18T23:33:44.285+0300 I STORAGE [initandlisten] Dropping unknown ident: index-5-7687852694704450203 2018-01-18T23:33:44.309+0300 I STORAGE [initandlisten] Dropping unknown ident: index-6-7687852694704450203 2018-01-18T23:33:44.331+0300 I STORAGE [initandlisten] Dropping unknown ident: index-7-7687852694704450203 2018-01-18T23:33:44.353+0300 I STORAGE [initandlisten] Dropping unknown ident: index-8-7687852694704450203 2018-01-18T23:33:44.379+0300 I STORAGE [initandlisten] Dropping unknown ident: index-9-7687852694704450203 2018-01-18T23:33:44.397+0300 I STORAGE [initandlisten] finished checking dbs 2018-01-18T23:33:44.397+0300 I NETWORK [initandlisten] shutdown: going to close listening sockets... 2018-01-18T23:33:44.398+0300 I NETWORK [initandlisten] removing socket file: /tmp/mongodb-27017.sock 2018-01-18T23:33:44.398+0300 I STORAGE [initandlisten] WiredTigerKVEngine shutting down 2018-01-18T23:33:44.458+0300 I STORAGE [initandlisten] shutdown: removing fs lock... 2018-01-18T23:33:44.458+0300 I CONTROL [initandlisten] now exiting 2018-01-18T23:33:44.458+0300 I CONTROL [initandlisten] shutting down with code:0 there is no replica server or working backup unfortunately I'm sharing files bellow any help would be really appreciated. Regards

Top User Comments

thomas.schubert commented on Fri, 16 Feb 2018 22:37:38 +0000: Hi langston, We haven’t heard back from you for some time, so I’m going to mark this ticket as resolved. If this is still an issue for you, please provide additional information about your data files and we will reopen the ticket. Regards, Kelsey thomas.schubert commented on Tue, 23 Jan 2018 04:22:29 +0000: Hi langston, After reviewing the files, it appears that the _mdb_catalog.wt suffered from some form of os storage layer corruption. Unfortunately, during repair, this file is considered the authoritative list of collections and indexes stored by WiredTiger. Since these data files in the WT Catalog were not referenced by the corrupted _mdb_catalog.wt, the repair process dropped the files to free disk space taken up by data that could never be accessed. We're looking into modifying the behavior of --repair to not drop unknown idents, but either move them aside or preferably recreate catalog entries for them (in a lost+found database or such) and let the user explicitly delete them again if they're unwanted. In terms of data recovery, it is not possible at this time to determine whether only _mdb_catalog.wt was affected or if there is corruption on other files as well. If you have a backup prior to running the repair operation, we can attempt a manual repair using these files. If not, I would suggest taking a backup of your current files, and then run a filesystem recovery tool to recover the data files using a recovery software or service. If you're able to recover these files, please let us know so we can attempt to help repair this corruption. For us to attempt to manually recover data, we would need a complete directory listing of the $dbpath as well as information about options that were set when creating these collections. Additionally, would you please clarify when this mongod was always running 3.6.1? If the collections were created in on an earlier version, would you please specify and clarify whether the Feature Compatibility was upgraded to 3.6? If the size of your database is not unwieldly, I would be happy to provide a secure upload portal as it may be help us if you could provide your complete $dbpath, but I don't think it's necessary for us to proceed. Thank you, Kelsey ramon.fernandez commented on Fri, 19 Jan 2018 19:34:32 +0000: Sorry to hear you've run into issues langston, we're looking into this issue and the files you uploaded, and will post updates in this ticket as we have them. Regards, Ramón.

Steps to Reproduce