Info

Hello! Our MongoDB base is corrupted by problems with hard drive. We recover our disk with database and MongoDB server We check config and MongoDB database folder. Config ok and have a right dbpath. In dbpath directory we have many files like this: total 2.5G rwxrwxr- 1 mongodb mongodb 21 Apr 16 2017 WiredTiger.lock rwxrwxr- 1 mongodb mongodb 95 Apr 16 2017 storage.bson rwxrwxr- 1 mongodb mongodb 16K Apr 16 2017 index-4--2017253496148277652.wt rwxrwxr- 1 mongodb mongodb 52K Oct 10 2017 index-15-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 36K May 20 02:33 _mdb_catalog.wt.1 rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 collection-2--2017253496148277652.wt rwxrwxr- 1 mongodb mongodb 36K May 20 02:33 collection-2--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 72K May 20 02:33 collection-18-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 52K May 20 02:33 collection-14-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 100K May 20 02:33 collection-0--4753795739824518231.wt rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 index-3--2017253496148277652.wt rwxrwxr- 1 mongodb mongodb 16K May 20 02:33 index-3--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 36K May 20 02:34 index-1--2017253496148277652.wt rwxrwxr- 1 mongodb mongodb 40K May 20 02:34 collection-0--2017253496148277652.wt rwxrwxr- 1 mongodb mongodb 44K May 20 02:38 index-19-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 16K May 20 02:38 index-1--4753795739824518231.wt rwxrwxr- 1 mongodb mongodb 40K May 20 02:51 index-11-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 16K May 20 06:49 index-1-7644913279549684036.wt rwxrwxr- 1 mongodb mongodb 32K Jun 5 11:53 index-3-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 36K Jun 5 11:53 collection-2-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 52K Jun 17 08:26 index-13-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 132K Jun 22 10:14 collection-12-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 68K Aug 8 03:35 index-25-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 322M Aug 8 03:35 collection-24-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 36K Aug 13 15:24 index-9--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 60K Aug 13 15:24 collection-8--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 368K Aug 15 10:06 index-21-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 1.2M Aug 15 10:06 collection-20-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 84K Aug 16 10:23 index-7-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 192K Aug 16 10:23 collection-6-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 772K Aug 16 14:49 index-5-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 148K Aug 16 14:49 index-23-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 6.1M Aug 16 14:49 collection-4-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 1.1M Aug 16 14:49 collection-22-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 324K Aug 16 14:49 collection-10-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 36K Aug 16 14:49 collection-0-7644913279549684036.wt rwxrwxr- 1 mongodb mongodb 2.2M Aug 17 11:50 index-17-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 16M Aug 17 11:50 collection-16-6056211751025971878.wt rwxrwxr- 1 mongodb mongodb 468K Aug 17 16:59 index-7--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 864K Aug 17 16:59 collection-6--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 23M Aug 18 00:16 index-1--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 2.1G Aug 18 00:16 collection-0--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 968K Aug 18 00:17 index-5--3777467847017729897.wt rwxrwxr- 1 mongodb mongodb 124K Aug 18 00:59 WiredTiger.wt.1 rwxrwxr- 1 mongodb mongodb 49 Aug 28 09:12 WiredTiger rwxrwxr- 1 mongodb mongodb 1.6M Aug 28 09:12 collection-4--3777467847017729897.wt rw-rr- 1 mongodb nogroup 16K Aug 28 09:37 index-1-7056954381577752709.wt rwxrwxr- 1 mongodb mongodb 6 Aug 28 09:38 mongod.lock drwxrwxr-- 2 mongodb mongodb 4.0K Aug 28 09:39 journal rw-rr- 1 mongodb nogroup 1000 Aug 28 09:40 WiredTiger.turtle drwxrwxr-- 2 mongodb mongodb 4.0K Aug 28 10:54 diagnostic.data rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 _mdb_catalog.wt rw-rr- 1 mongodb nogroup 32K Aug 28 10:55 index-2-7056954381577752709.wt rw-rr- 1 mongodb nogroup 16K Aug 28 10:55 index-0-7056954381577752709.wt rwxrwxr- 1 mongodb mongodb 16K Aug 28 10:55 collection-2-3814419529578137210.wt rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 collection-0-3814419529578137210.wt rwxrwxr- 1 mongodb mongodb 92K Aug 28 10:55 WiredTiger.wt rw-rr- 1 mongodb nogroup 4.0K Aug 28 10:55 WiredTigerLAS.wt rwxrwxr- 1 mongodb mongodb 36K Aug 28 10:55 sizeStorer.wt Its looks like a database! Our mongod daemon started and we can enter in it by Robo3t, but we see only admin and local databases. We try to repair it by "mongod --repair --dbpath (path)" key and its nothing gave to as We try "./wt -v -h ../mongo-bak -C "extensions=[./ext/compressors/snappy/.libs/libwiredtiger_snappy.so]" -R salvage collection-2657-1723320556100349955.wt" on all collections in dbpath directory (by WiredTiger utility) and its nothing gave to as again Can you help as with this problem? Thanks! Have a good day!

Top User Comments

it@iidf.ru commented on Fri, 31 Aug 2018 07:53:52 +0000: Ok! Thank you for your help, Nick! Have a good day! =) nick.brewer commented on Thu, 30 Aug 2018 20:15:06 +0000: it@iidf.ru This still appears to be a permissions issue - I would suggest ensuring that all of the files within /var/lib/mongodb are owned by the mongodb user and group, and then removing the mongod.lock file, and rebooting the mongod. Since this doesn't demonstrate an underlying bug in MongoDB, I'm going to close this ticket. For MongoDB-related support discussion I suggest posting on the mongodb-user group or Stack Overflow with the mongodb tag. A question like this involving more discussion would be best posted on the mongodb-user group. -Nick it@iidf.ru commented on Thu, 30 Aug 2018 06:56:07 +0000: Hello! When we try to "systemctl  start mongod" we get this message: root@db01:~# systemctl start mongod root@db01:~# systemctl status mongod ● mongod.service - High-performance, schema-free document-oriented database Loaded: loaded (/etc/systemd/system/mongod.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Thu 2018-08-30 02:50:08 EDT; 4s ago Process: 4126 ExecStart=/usr/bin/mongod --verbose --config /etc/mongod.conf (code=exited, status=1/FAILURE) Main PID: 4126 (code=exited, status=1/FAILURE) Aug 30 02:50:08 db01 systemd[1]: Started High-performance, schema-free document-oriented database. Aug 30 02:50:08 db01 systemd[1]: mongod.service: Main process exited, code=exited, status=1/FAILURE Aug 30 02:50:08 db01 systemd[1]: mongod.service: Unit entered failed state. Aug 30 02:50:08 db01 systemd[1]: mongod.service: Failed with result 'exit-code'. In log we get: 2018-08-30T02:53:01.977-0400 I CONTROL [main] ***** SERVER RESTARTED ***** 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] MongoDB starting : pid=4196 port=27017 dbpath=/var/lib/mongodb 64-bit host=db01 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] db version v3.4.3 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] git version: f07437fb5a6cca07c10bafa78365456eb1d6d5e1 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.1t 3 May 2016 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] allocator: tcmalloc 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] modules: none 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] build environment: 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] distmod: debian81 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] distarch: x86_64 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] target_arch: x86_64 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] options: { config: "/etc/mongod.conf", net: { bindIp: "127.0.0.1,192.168.70.14", port: 27017 } , storage: { dbPath: "/var/lib/mongodb", journal: { enabled: true } }, systemLog: { destination: "file", logAppend: true, path: "/var/log/mongodb/mongod.log", verbosity: 1 } } 2018-08-30T02:53:01.981-0400 D - [initandlisten] User Assertion: 28596:Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied: "/var/lib/mongodb/mongod.lock" src/mongo/db/service_context_d.cpp 86 2018-08-30T02:53:01.981-0400 I STORAGE [initandlisten] exception in initAndListen: 28596 Unable to determine status of lock file in the data directory /var/lib/mongodb: boost::filesystem::status: Permission denied: "/var/lib/mongodb/mongod.lock", terminating 2018-08-30T02:53:01.981-0400 I NETWORK [initandlisten] shutdown: going to close listening sockets... 2018-08-30T02:53:01.981-0400 I NETWORK [initandlisten] shutdown: going to flush diaglog... 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] now exiting 2018-08-30T02:53:01.981-0400 I CONTROL [initandlisten] shutting down with code:100 I see "Permission denied" message, but I check permissions of the dbpath directory, it's ok =( nick.brewer commented on Thu, 30 Aug 2018 01:54:39 +0000: it@iidf.ru Are you starting the mongod in a different way than you normally would? For example, do you usually start mongod via the init system (something like sudo systemctl start mongod or sudo service mongod start), and are you instead starting it manually via mongod --repair --dbpath (path)? If so, I'd like to know what happens when you try starting the mongod as you normally would. Thanks, Nick it@iidf.ru commented on Wed, 29 Aug 2018 07:08:15 +0000: Hello! Yes, we make a "chmod R 664 /var/lib/mongodb" and "chown -R mongodb:mongodb /var/lib/mongodb" before start the daemon I make new logs for you And I can't find the "src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp" directory, where it must been? nick.brewer commented on Tue, 28 Aug 2018 19:49:20 +0000: it@iidf.ru From the log file you've provided, it appears that you're running into a permissions issue after the repair you attempted: Assertion: 28595:13: Permission denied src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 267 Have you compared the permissions of the files in your dbpath after the attempt to repair with the wt binary against the permissions of the files in the backup taken last week? -Nick it@iidf.ru commented on Tue, 28 Aug 2018 18:53:13 +0000: From the MongoDB shell we see 2 standart databases (admin and local) and a standart collections in it. In sum we see with shell the some thing like a with Robo3T it@iidf.ru commented on Tue, 28 Aug 2018 18:47:18 +0000: MongoDB version is 3.4.3 Linux db01 3.16.0-4-amd64 #1 SMP Debian 3.16.39-1+deb8u2 (2017-03-07) x86_64 GNU/Linux Server working at VMvare as virtual machine We have a backup by 1 week ago, we try to put it instead of original database folder and result not change =( We have a WiredTiger.wt and WiredTiger.wt.1, I upload both to this issue Also I upload a log at disk breakdown moment. Thank you for your help! nick.brewer commented on Tue, 28 Aug 2018 18:29:46 +0000: it@iidf.ru Can you include the log entries from the time you start the mongod? Additionally, have you checked what collections are available via the mongo shell directly, as a user with appropriate access? I should note that we don't recommend attempting to recover files via the wt binary directly. If you have any backups available from before the hard drive issue, this would likely be the fastest way to get your database back into a functional state. That said, if you can upload your WiredTiger.wt and WiredTiger.turtle files, I would be happy to attempt a repair; in addition to the requested log entry, I'd need to confirm: The MongoDB version The operating system and version Platform (virtual machine, container, native hardware, etc) Thanks, Nick

Steps to Reproduce