Info

After a (clean) server shutdown/reboot our MongoDB-Server fails to start with the follow messages: 2017-06-26T12:11:28.491+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1498471888:491849][14448:0x7f796a509cc0], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error 2017-06-26T12:11:28.491+0200 E STORAGE [initandlisten] WiredTiger error (0) [1498471888:491897][14448:0x7f796a509cc0], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata 2017-06-26T12:11:28.491+0200 E STORAGE [initandlisten] WiredTiger error (0) [1498471888:491907][14448:0x7f796a509cc0], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk 2017-06-26T12:11:28.491+0200 E STORAGE [initandlisten] WiredTiger error (0) [1498471888:491916][14448:0x7f796a509cc0], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options 2017-06-26T12:11:28.492+0200 I - [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267 We've tried different repair attempts but all of them failed. In other tickets we've seen that your may are able to repair the corrupt files. Any help would be appriciated. If you need any further details or files i'll provide them as soon as possible. Thanks in advance.

Top User Comments

dracoblue commented on Wed, 28 Jun 2017 15:29:22 +0000: Thank you! It's working again! dracoblue commented on Tue, 27 Jun 2017 20:05:39 +0000: Hey Mark, I work at Exozet and I am happy about your quick response! A quick test on a docker container with mongodb 3.4.1 and the dataset, showed this response: 2017-06-27T19:56:20.820+0000 I CONTROL [initandlisten] MongoDB starting : pid=7 port=27017 dbpath=/data/db 64-bit host=b2a8c66ec9c3 2017-06-27T19:56:20.820+0000 I CONTROL [initandlisten] db version v3.4.1 2017-06-27T19:56:20.820+0000 I CONTROL [initandlisten] git version: 5e103c4f5583e2566a45d740225dc250baacfbd7 2017-06-27T19:56:20.820+0000 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016 2017-06-27T19:56:20.820+0000 I CONTROL [initandlisten] allocator: tcmalloc 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] modules: none 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] build environment: 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] distmod: debian81 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] distarch: x86_64 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] target_arch: x86_64 2017-06-27T19:56:20.821+0000 I CONTROL [initandlisten] options: {} 2017-06-27T19:56:20.831+0000 I - [initandlisten] Detected data files in /data/db created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'. 2017-06-27T19:56:20.833+0000 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=1461M,session_max=20000,eviction=(threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0), 2017-06-27T19:56:21.504+0000 E STORAGE [initandlisten] WiredTiger error (0) [1498593381:504600][7:0x7f832fd79cc0], file:index-3--6053467142352765237.wt, txn-recover: read checksum error for 4096B block at offset 806912: block header checksum of 1902734456 doesn't match expected checksum of 2689960430 2017-06-27T19:56:21.504+0000 E STORAGE [initandlisten] WiredTiger error (0) [1498593381:504676][7:0x7f832fd79cc0], file:index-3--6053467142352765237.wt, txn-recover: index-3--6053467142352765237.wt: encountered an illegal file format or internal value 2017-06-27T19:56:21.504+0000 E STORAGE [initandlisten] WiredTiger error (-31804) [1498593381:504698][7:0x7f832fd79cc0], file:index-3--6053467142352765237.wt, txn-recover: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-06-27T19:56:21.504+0000 I - [initandlisten] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361 2017-06-27T19:56:21.504+0000 I - [initandlisten] ***aborting after fassert() failure 2017-06-27T19:56:21.532+0000 F - [initandlisten] Got signal: 6 (Aborted). 0x55e956be32c1 0x55e956be23b9 0x55e956be289d 0x7f832e9d5890 0x7f832e650067 0x7f832e651448 0x55e955e7deed 0x55e9568fe626 0x55e955e88219 0x55e955e8830d 0x55e955e88565 0x55e9574e3aef 0x55e9574e403b 0x55e9574e074d 0x55e9574e5217 0x55e9575028c6 0x55e957538c1b 0x55e9575c67c1 0x55e9575c6ed0 0x55e9575c740c 0x55e957548a99 0x55e9575bb400 0x55e9575d9850 0x55e9575d9c85 0x55e95756ddd9 0x55e9575da7f2 0x55e95753ccb7 0x55e9575350ae 0x55e9568e1c56 0x55e9568daaec 0x55e9567ca1b0 0x55e955e698e3 0x55e955e8984f 0x7f832e63cb45 0x55e955ee73ff ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55E95566B000","o":"15782C1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55E95566B000","o":"15773B9"},{"b":"55E95566B000","o":"157789D"},{"b":"7F832E9C6000","o":"F890"},{"b":"7F832E61B000","o":"35067","s":"gsignal"},{"b":"7F832E61B000","o":"36448","s":"abort"},{"b":"55E95566B000","o":"812EED","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"55E95566B000","o":"1293626"},{"b":"55E95566B000","o":"81D219","s":"__wt_eventv"},{"b":"55E95566B000","o":"81D30D","s":"__wt_err"},{"b":"55E95566B000","o":"81D565","s":"__wt_panic"},{"b":"55E95566B000","o":"1E78AEF","s":"__wt_block_extlist_read"},{"b":"55E95566B000","o":"1E7903B","s":"__wt_block_extlist_read_avail"},{"b":"55E95566B000","o":"1E7574D","s":"__wt_block_checkpoint_load"},{"b":"55E95566B000","o":"1E7A217"},{"b":"55E95566B000","o":"1E978C6","s":"__wt_btree_open"},{"b":"55E95566B000","o":"1ECDC1B","s":"__wt_conn_btree_open"},{"b":"55E95566B000","o":"1F5B7C1","s":"__wt_session_get_btree"},{"b":"55E95566B000","o":"1F5BED0","s":"__wt_session_get_btree"},{"b":"55E95566B000","o":"1F5C40C","s":"__wt_session_get_btree_ckpt"},{"b":"55E95566B000","o":"1EDDA99","s":"__wt_curfile_open"},{"b":"55E95566B000","o":"1F50400"},{"b":"55E95566B000","o":"1F6E850"},{"b":"55E95566B000","o":"1F6EC85"},{"b":"55E95566B000","o":"1F02DD9","s":"__wt_log_scan"},{"b":"55E95566B000","o":"1F6F7F2","s":"__wt_txn_recover"},{"b":"55E95566B000","o":"1ED1CB7","s":"__wt_connection_workers"},{"b":"55E95566B000","o":"1ECA0AE","s":"wiredtiger_open"},{"b":"55E95566B000","o":"1276C56","s":"_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb"},{"b":"55E95566B000","o":"126FAEC"},{"b":"55E95566B000","o":"115F1B0","s":"_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv"},{"b":"55E95566B000","o":"7FE8E3"},{"b":"55E95566B000","o":"81E84F","s":"main"},{"b":"7F832E61B000","o":"21B45","s":"__libc_start_main"},{"b":"55E95566B000","o":"87C3FF"}],"processInfo":{ "mongodbVersion" : "3.4.1", "gitVersion" : "5e103c4f5583e2566a45d740225dc250baacfbd7", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.9.27-moby", "version" : "#1 SMP Thu May 11 04:01:18 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "55E95566B000", "elfType" : 3, "buildId" : "38074E101F25D703C8F60A85C8471A7AE340C023" }, { "b" : "7FFCBC139000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "014FE5C80CE888426B90D5FA477DB6C4B0AC5D11" }, { "b" : "7F832F902000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "0711D67B299ACA3EA6B0B57C0DA4EC83B71787EE" }, { "b" : "7F832F506000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "2E613A282B22F2B0978ACCEC1B71411F5E20CC89" }, { "b" : "7F832F2FE000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "906B9D78305E46BC76994F552FA63751C51CD065" }, { "b" : "7F832F0FA000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "EDDA40FF0B16D74E776AEA74FAAE6B898ACD2D15" }, { "b" : "7F832EDF9000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "1B8F009691E3224A991F1F6517A74DA30A065B9A" }, { "b" : "7F832EBE3000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "D5FB04F64B3DAEA6D6B68B5E8B9D4D2BC1A6E1FC" }, { "b" : "7F832E9C6000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "1ADC4ADBA1D853EEA9A5B3CD49E25AF85DCA0100" }, { "b" : "7F832E61B000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "7A02D454BA0E8AF69E3A284C381318B55908DEDA" }, { "b" : "7F832FB63000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "060BF28EEE293312DDF82D4DBEF40B3BA8927F0A" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55e956be32c1] mongod(+0x15773B9) [0x55e956be23b9] mongod(+0x157789D) [0x55e956be289d] libpthread.so.0(+0xF890) [0x7f832e9d5890] libc.so.6(gsignal+0x37) [0x7f832e650067] libc.so.6(abort+0x148) [0x7f832e651448] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x55e955e7deed] mongod(+0x1293626) [0x55e9568fe626] mongod(__wt_eventv+0x427) [0x55e955e88219] mongod(__wt_err+0x9D) [0x55e955e8830d] mongod(__wt_panic+0x24) [0x55e955e88565] mongod(__wt_block_extlist_read+0x8F) [0x55e9574e3aef] mongod(__wt_block_extlist_read_avail+0x2B) [0x55e9574e403b] mongod(__wt_block_checkpoint_load+0x27D) [0x55e9574e074d] mongod(+0x1E7A217) [0x55e9574e5217] mongod(__wt_btree_open+0xBC6) [0x55e9575028c6] mongod(__wt_conn_btree_open+0x15B) [0x55e957538c1b] mongod(__wt_session_get_btree+0x101) [0x55e9575c67c1] mongod(__wt_session_get_btree+0x810) [0x55e9575c6ed0] mongod(__wt_session_get_btree_ckpt+0x14C) [0x55e9575c740c] mongod(__wt_curfile_open+0x179) [0x55e957548a99] mongod(+0x1F50400) [0x55e9575bb400] mongod(+0x1F6E850) [0x55e9575d9850] mongod(+0x1F6EC85) [0x55e9575d9c85] mongod(__wt_log_scan+0xAE9) [0x55e95756ddd9] mongod(__wt_txn_recover+0x582) [0x55e9575da7f2] mongod(__wt_connection_workers+0x37) [0x55e95753ccb7] mongod(wiredtiger_open+0x175E) [0x55e9575350ae] mongod(_ZN5mongo18WiredTigerKVEngineC2ERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEES8_PNS_11ClockSourceES8_mbbbb+0xA76) [0x55e9568e1c56] mongod(+0x126FAEC) [0x55e9568daaec] mongod(_ZN5mongo20ServiceContextMongoD29initializeGlobalStorageEngineEv+0x6E0) [0x55e9567ca1b0] mongod(+0x7FE8E3) [0x55e955e698e3] mongod(main+0x95F) [0x55e955e8984f] libc.so.6(__libc_start_main+0xF5) [0x7f832e63cb45] mongod(+0x87C3FF) [0x55e955ee73ff] ----- END BACKTRACE ----- Aborted Does this help you to track down the issue? [edit] I build an docker image with wiredtiger 1.8.0 wt cli tool and ran salvage command on the index checksum file in question and was able to launch mongodb right after that again. Will test tomorrow, if the database works as expected again on our production environment. mark.agarunov commented on Tue, 27 Jun 2017 19:45:58 +0000: Hello xoz, Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. Additionally, if this issue persists, please provide the following information: What kind of underlying storage mechanism are you using? Are the storage devices attached locally or over the network? Are the disks SSDs or HDDs? What kind of RAID and/or volume management system are you using? Would you please check the integrity of your disks? Has the database always been running this version of MongoDB? If not please describe the upgrade/downgrade cycles the database has been through. Have you manipulated (copied or moved) the underlying database files? If so, was mongod running? Have you ever restored this instance from backups? What method do you use to create backups? When was the underlying filesystem last checked and is it currently marked clean? Thanks, Mark

Steps to Reproduce