Info

Last night our mongodb server was crashed with the error message " E STORAGE [WTCheckpointThread] WiredTiger error (0) [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293) Raw: [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (_wt_turtle_read, 293)" Then I tried to restart the server and also run the mongod --repair command but I always get the same error. The log file and wire tiger files are attached. Thanks in advance for any help

Top User Comments

nick.brewer commented on Mon, 1 Oct 2018 20:16:22 +0000: afshan886@gmail.com Have you had a chance to look into whether another process may be manipulating dbpath files while the mongod is running? Thanks, -Nick nick.brewer commented on Fri, 14 Sep 2018 16:55:33 +0000: afshan886@gmail.com Glad to hear you were able to get it working from a backup. Looking at your logs, it appears that you've seen a few different errors in the past related to the WiredTiger.turtle file, some of them permissions-based, and some of them most likely due to corruption: 2018-08-27T21:58:52.084+0200 E STORAGE [WTCheckpointThread] WiredTiger error (103) [1535399932:84754][463:0x7fe33e567700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle.set: handle-open: open: Software caused connection abort Raw: [1535399932:84754][463:0x7fe33e567700], file:WiredTiger.wt, WT_SESSION.checkpoint: /var/lib/mongodb/WiredTiger.turtle.set: handle-open: open: Software caused connection abort 2018-09-06T23:56:50.760+0200 E STORAGE [WTCheckpointThread] WiredTiger error (0) [1536271010:760909][453:0x7f57298e9700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value: (__wt_turtle_read, 293) 2018-08-16T12:23:46.219+0200 E STORAGE [initandlisten] WiredTiger error (13) [1534415026:219001][4273:0x7f33e59029c0], wiredtiger_open: /var/lib/mongodb/WiredTiger.turtle: handle-open: open: Permission denied I'm curious if there is another process that may be attempting to manipulate these files while the mongod is running? Thanks, -Nick afshan886@gmail.com commented on Fri, 14 Sep 2018 08:41:25 +0000: nick.brewer Thanks for your reply. Okay I restored the database from a backup. The answers to your questions are: the cause of the issue was unexpected shutdown the underlying platform is virtual machine Thanks again nick.brewer commented on Tue, 11 Sep 2018 16:15:47 +0000: afshan886@gmail.com It appears that your WiredTiger.turtle file is badly corrupted - as this file contains metadata that is used to interpret other WiredTiger files, it will not be possible to proceed with a repair using the file in this state. If you have a backup of your dbpath available I would recommend that you restore from it to get you database into working condition. Some other things I'd like to confirm: What was the cause of this issue (power outage, unexpected shutdown, filesystem issues, etc)? What is the underlying platform (virtual machine, container, native hardware, etc)? Thanks, -Nick

Steps to Reproduce