Info

Hi , Our production environment is down because monogodb is not starting and repair option also is failing. The log says Cannot start mongod or --repair because of WiredTiger.wt checksum error 2017-06-02T05:24:40.881-0700 E STORAGE [initandlisten] WiredTiger (0) [1496406280:881822][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: read checksum error for 4096B block at offset 192512: block header checksum of 1768714355 doesn't match expected checksum of 2158401612 2017-06-02T05:24:40.881-0700 E STORAGE [initandlisten] WiredTiger (0) [1496406280:881959][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: WiredTiger.wt: encountered an illegal file format or internal value 2017-06-02T05:24:40.882-0700 E STORAGE [initandlisten] WiredTiger (-31804) [1496406280:882145][8955:0x7fcfd8daeb80], file:WiredTiger.wt, connection: the process must exit and restart: WT_PANIC: WiredTiger library panic 2017-06-02T05:24:40.882-0700 I - [initandlisten] Fatal Assertion 28558 2017-06-02T05:24:40.900-0700 I CONTROL [initandlisten] 0xf9cfe2 0xf3a269 0xf1e416 0xdbf331 0x13f20cc 0x13f227d 0x13f26f4 0x133d07c 0x133d5d3 0x133a1f2 0x133e31e 0x13592f4 0x138481b 0x13f123b 0x13bdce3 0x1381d89 0xda956b 0xda7140 0xaa84ba 0x8265a1 0x7f42d9 0x7fcfd738876d 0x8242ad ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"B9CFE2"},{"b":"400000","o":"B3A269"},{"b":"400000","o":"B1E416"},{"b":"400000","o":"9BF331"},{"b":"400000","o":"FF20CC"},{"b":"400000","o":"FF227D"},{"b":"400000","o":"FF26F4"},{"b":"400000","o":"F3D07C"},{"b":"400000","o":"F3D5D3"},{"b":"400000","o":"F3A1F2"},{"b":"400000","o":"F3E31E"},{"b":"400000","o":"F592F4"},{"b":"400000","o":"F8481B"},{"b":"400000","o":"FF123B"},{"b":"400000","o":"FBDCE3"},{"b":"400000","o":"F81D89"},{"b":"400000","o":"9A956B"},{"b":"400000","o":"9A7140"},{"b":"400000","o":"6A84BA"},{"b":"400000","o":"4265A1"},{"b":"400000","o":"3F42D9"},{"b":"7FCFD7367000","o":"2176D"},{"b":"400000","o":"4242AD"}],"processInfo":{ "mongodbVersion" : "3.0.8", "gitVersion" : "83d8cc25e00e42856924d84e220fbe4a839e605d", "uname" : { "sysname" : "Linux", "release" : "3.13.0-37-generic", "version" : "#64~precise1-Ubuntu SMP Wed Sep 24 21:37:11 UTC 2014", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "BDDA1C988F0ABFD36E0CF92E44313557B4CDCF59" }, { "b" : "7FFF238FE000", "elfType" : 3, "buildId" : "6C2D7ED95200205479A32E29DEE3CD929E125B36" }, { "b" : "7FCFD897E000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "CF53EC11DE0453E16A0021DECCC5AE8DFF132044" }, { "b" : "7FCFD871F000", "path" : "/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "42220C46A8D4CC0B9293D4CDF5708D6C37E3A867" }, { "b" : "7FCFD8343000", "path" : "/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "3181CF2B60453D363A56E8A78F6F9493E86F2B77" }, { "b" : "7FCFD813B000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "C9D0D71097C68F201557FA08EED37724F8A10197" }, { "b" : "7FCFD7F37000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "FD9488B6A172ED4E8FFDD32F6A8F9B52AFE3BD33" }, { "b" : "7FCFD7C37000", "path" : "/usr/lib/x86_64-linux-gnu/libstdc++.so.6", "elfType" : 3, "buildId" : "B534DA725D06A04267EB2FEB92B9CC14C838B57B" }, { "b" : "7FCFD793B000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "BA13740381D9E8D88BD16A2EF8834F9E406D8542" }, { "b" : "7FCFD7725000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "ECF322A96E26633C5D10F18215170DD4395AF82C" }, { "b" : "7FCFD7367000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "AC247F95A3044C7F6C4C1F17E14B7FF06BDB2B64" }, { "b" : "7FCFD8B9B000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "81065F00F6E667ABED8F4AF6CFA87DA0D01693A9" }, { "b" : "7FCFD7150000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "F695ECFCF3918D5D34989398A14B7ECDD9F46CD0" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x32) [0xf9cfe2] mongod(_ZN5mongo10logContextEPKc+0xE9) [0xf3a269] mongod(_ZN5mongo13fassertFailedEi+0x66) [0xf1e416] mongod(+0x9BF331) [0xdbf331] mongod(__wt_eventv+0x49C) [0x13f20cc] mongod(__wt_err+0x8D) [0x13f227d] mongod(__wt_panic+0x24) [0x13f26f4] mongod(__wt_block_extlist_read+0x6C) [0x133d07c] mongod(__wt_block_extlist_read_avail+0x33) [0x133d5d3] mongod(__wt_block_checkpoint_load+0x202) [0x133a1f2] mongod(+0xF3E31E) [0x133e31e] mongod(__wt_btree_open+0xB54) [0x13592f4] mongod(__wt_conn_btree_get+0x1AB) [0x138481b] mongod(__wt_session_get_btree+0x3EB) [0x13f123b] mongod(__wt_metadata_open+0x33) [0x13bdce3] mongod(wiredtiger_open+0xD99) [0x1381d89] mongod(_ZN5mongo18WiredTigerKVEngineC1ERKSsS2_bb+0x53B) [0xda956b] mongod(+0x9A7140) [0xda7140] mongod(_ZN5mongo23GlobalEnvironmentMongoD22setGlobalStorageEngineERKSs+0x32A) [0xaa84ba] mongod(_ZN5mongo13initAndListenEi+0x421) [0x8265a1]

Top User Comments

thomas.schubert commented on Fri, 16 Jun 2017 13:24:25 +0000: Hi sneha_k_a, I've examined the provided log files. Unfortunately, this error indicates that the disk corruption affected more than just the WiredTiger.wt file. In this situation, I would recommend checking the integrity of your disks and then restoring from a backup. Kind regards, Thomas sneha_k_a commented on Fri, 16 Jun 2017 08:47:12 +0000: Hi Thomas, Can we get an update on this. Regards Sneha sneha_k_a commented on Tue, 13 Jun 2017 20:40:58 +0000: Thanks for the reply Thomas. On replacing the files provided by you in $dbpath, the above mentioned error is gone but I am running into another issue. Hence I am attaching the mongod.log as requested by you. thomas.schubert commented on Tue, 13 Jun 2017 18:37:13 +0000: Hi sneha_k_a, Thank you for the report. I've attached a repair attempt of the files you've provided. Would you please extract these files and replace them in your $dbpath and let us know if it resolves the issue? If you are still seeing errors after replacing these files, please provide the complete logs from mongod so that we can further investigate. While this issue is likely related to some sort of disk failure, I would recommend upgrading to the latest version of either MongoDB 3.2 or 3.4 to take advantage of significant improvements to WiredTiger. Kind regards, Thomas sneha_k_a commented on Tue, 13 Jun 2017 18:29:50 +0000: Attaching the required attachments.

Steps to Reproduce