Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Hello Team, I am testing MongoDB 4.0 on my local Ubuntu 18.04.1 and mongoDB crashed when I insert document with following error. Log file attached 2018-07-30T20:15:48.153+0000 E STORAGE [thread3] WiredTiger error (0) [1532981748:153737][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: encountered an illegal file format or internal value: (__inmem_row_leaf, 617) Raw: [1532981748:153737][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: encountered an illegal file format or internal value: (__inmem_row_leaf, 617) 2018-07-30T20:15:48.153+0000 E STORAGE [thread3] WiredTiger error (-31804) [1532981748:153796][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic Raw: [1532981748:153796][2126:0x7f81821f0700], file:index-11--1602984826302750418.wt, eviction-server: the process must exit and restart: WT_PANIC: WiredTiger library panic 2018-07-30T20:15:48.153+0000 F - [thread3] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403 2018-07-30T20:15:48.153+0000 F - [thread3] ***aborting after fassert() failure 2018-07-30T20:15:48.153+0000 F - [conn2] Fatal Assertion 28559 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 88 2018-07-30T20:15:48.153+0000 F - [conn2] ***aborting after fassert() failure 2018-07-30T20:15:48.154+0000 E STORAGE [thread4] WiredTiger error (-31804) [1532981748:154320][2126:0x7f81809ed700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic Raw: [1532981748:154320][2126:0x7f81809ed700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic 2018-07-30T20:15:48.154+0000 F - [thread4] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403 2018-07-30T20:15:48.154+0000 F - [thread4] ***aborting after fassert() failure 2018-07-30T20:15:48.172+0000 E STORAGE [thread5] WiredTiger error (-31804) [1532981748:172184][2126:0x7f81819ef700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic Raw: [1532981748:172184][2126:0x7f81819ef700], eviction-server: cache eviction thread error: WT_PANIC: WiredTiger library panic 2018-07-30T20:15:48.172+0000 F - [thread5] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403 2018-07-30T20:15:48.172+0000 F - [thread5] ***aborting after fassert() failure 2018-07-30T20:15:48.203+0000 E STORAGE [thread6] WiredTiger error (-31804) [1532981748:203320][2126:0x7f81829f1700], log-server: log server error: WT_PANIC: WiredTiger library panic Raw: [1532981748:203320][2126:0x7f81829f1700], log-server: log server error: WT_PANIC: WiredTiger library panic 2018-07-30T20:15:48.203+0000 F - [thread6] Fatal Assertion 50853 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 403 2018-07-30T20:15:48.203+0000 F - [thread6] ***aborting after fassert() failure 2018-07-30T20:15:48.485+0000 F - [thread3] Got signal: 6 (Aborted). 0x560f086dd141 0x560f086dc359 0x560f086dc83d 0x7f8189b03890 0x7f818973ee97 0x7f8189740801 0x560f06d5d22f 0x560f06e4e726 0x560f06ebc081 0x560f06ce8119 0x560f06ce8442 0x560f06ee710d 0x560f06ef3af2 0x560f06ef8cf0 0x560f06e7902a 0x560f06e716a3 0x560f06e71bd3 0x560f06e74731 0x560f06ec85e9 0x7f8189af86db 0x7f818982188f ----- BEGIN BACKTRACE ----- {"backtrace":[\{"b":"560F06365000","o":"2378141","s":"_ZN5mongo15printStackTraceERSo"},\{"b":"560F06365000","o":"2377359"},\{"b":"560F06365000","o":"237783D"},\{"b":"7F8189AF1000","o":"12890"},\{"b":"7F8189700000","o":"3EE97","s":"gsignal"},\{"b":"7F8189700000","o":"40801","s":"abort"},\{"b":"560F06365000","o":"9F822F","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},\{"b":"560F06365000","o":"AE9726"},\{"b":"560F06365000","o":"B57081"},\{"b":"560F06365000","o":"983119","s":"__wt_err"},\{"b":"560F06365000","o":"983442","s":"__wt_panic"},\{"b":"560F06365000","o":"B8210D","s":"__wt_page_inmem"},\{"b":"560F06365000","o":"B8EAF2"},\{"b":"560F06365000","o":"B93CF0","s":"__wt_split_rewrite"},\{"b":"560F06365000","o":"B1402A","s":"__wt_evict"},\{"b":"560F06365000","o":"B0C6A3"},\{"b":"560F06365000","o":"B0CBD3"},\{"b":"560F06365000","o":"B0F731","s":"__wt_evict_thread_run"},\{"b":"560F06365000","o":"B635E9"},\{"b":"7F8189AF1000","o":"76DB"},\{"b":"7F8189700000","o":"12188F","s":"clone"}],"processInfo":\{ "mongodbVersion" : "4.0.0", "gitVersion" : "3b07af3d4f471ae89e8186d33bbb1d5259597d51", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-29-generic", "version" : "#31-Ubuntu SMP Tue Jul 17 15:39:52 UTC 2018", "machine" : "x86_64" }, "somap" : [ \{ "b" : "560F06365000", "elfType" : 3, "buildId" : "365C0577E1D4FD5CD19A9455CB05F9122B9CF331" }, \{ "b" : "7FFFE4FF4000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "790546AE562C218C1F03B75290F2E817B4222315" }, \{ "b" : "7F818AF98000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "683A1F6A8C6401C61D29715C7F38B4586A175873" }, \{ "b" : "7F818AD7D000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, \{ "b" : "7F818A93A000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.0.0", "elfType" : 3, "buildId" : "5D934D586011BB3BD74719F877408B61FBD0AD3D" }, \{ "b" : "7F818A6D2000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.0.0", "elfType" : 3, "buildId" : "13EB7A057F4E3863403208D789D74A7CF58D66D5" }, \{ "b" : "7F818A4CE000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, \{ "b" : "7F818A2C6000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, \{ "b" : "7F8189F28000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, \{ "b" : "7F8189D10000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "039AE85FEF075EC14FE3528762A0645C8CF73B29" }, \{ "b" : "7F8189AF1000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, \{ "b" : "7F8189700000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, \{ "b" : "7F818B217000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, \{ "b" : "7F81894DB000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, \{ "b" : "7F81892BE000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, \{ "b" : "7F81890A2000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, \{ "b" : "7F8188E94000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, \{ "b" : "7F8188C49000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "451E2EC02C8F974B21D1CE47F5D2C21A21C16A9F" }, \{ "b" : "7F81889F7000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "0CDDF9D1F87FE7FF07D73DBE1D7CB57EE5BA802C" }, \{ "b" : "7F81887E9000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DD2893FC0AA3C58798F805D4C88EE086564D2EEB" }, \{ "b" : "7F81885CC000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, \{ "b" : "7F818824E000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, \{ "b" : "7F8187EE9000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, \{ "b" : "7F8187CB5000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, \{ "b" : "7F8187A7F000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, \{ "b" : "7F81877FE000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, \{ "b" : "7F8187528000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "AFF9A2A2B6ACB066BF450CC4A0772B9DC04299CB" }, \{ "b" : "7F81872F6000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "04AADCB371B97A570756D69A0D5B3B0E6737E349" }, \{ "b" : "7F81870F2000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "9E885FB7F54DA464A3D2A978C71C1178A6EC7512" }, \{ "b" : "7F8186EE7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "27A5983855C50A37EFA526BB12EA48CDAE3B5FCA" }, \{ "b" : "7F8186CCC000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, \{ "b" : "7F8186A8B000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, \{ "b" : "7F818675C000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, \{ "b" : "7F8186549000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, \{ "b" : "7F8186345000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, \{ "b" : "7F818613C000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, \{ "b" : "7F8185EAF000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, \{ "b" : "7F8185C0D000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, \{ "b" : "7F81859D7000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, \{ "b" : "7F81857C1000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, \{ "b" : "7F81855B9000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, \{ "b" : "7F8185390000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, \{ "b" : "7F8185181000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, \{ "b" : "7F8184F37000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, \{ "b" : "7F8184C2E000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, \{ "b" : "7F81849F6000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x560f086dd141] mongod(+0x2377359) [0x560f086dc359] mongod(+0x237783D) [0x560f086dc83d] libpthread.so.0(+0x12890) [0x7f8189b03890] libc.so.6(gsignal+0xC7) [0x7f818973ee97] libc.so.6(abort+0x141) [0x7f8189740801] mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x560f06d5d22f] mongod(+0xAE9726) [0x560f06e4e726] mongod(+0xB57081) [0x560f06ebc081] mongod(__wt_err+0x9D) [0x560f06ce8119] mongod(__wt_panic+0x33) [0x560f06ce8442] mongod(__wt_page_inmem+0x93D) [0x560f06ee710d] mongod(+0xB8EAF2) [0x560f06ef3af2] mongod(__wt_split_rewrite+0xA0) [0x560f06ef8cf0] mongod(__wt_evict+0x106A) [0x560f06e7902a] mongod(+0xB0C6A3) [0x560f06e716a3] mongod(+0xB0CBD3) [0x560f06e71bd3] mongod(__wt_evict_thread_run+0x71) [0x560f06e74731] mongod(+0xB635E9) [0x560f06ec85e9] libpthread.so.0(+0x76DB) [0x7f8189af86db] libc.so.6(clone+0x3F) [0x7f818982188f] ----- END BACKTRACE -----
nick.brewer commented on Mon, 1 Oct 2018 20:38:18 +0000: kelvinl Since there's been no activity on this ticket in some time, I'm going to close it. Feel free to update this ticket with the requested information, and we'll reopen it for you. Thanks, -Nick nick.brewer commented on Mon, 20 Aug 2018 23:41:35 +0000: kelvinl Could provide the following information, if possible: The full mongod logs leading up to the crash Any details required to recreate this behavior (a specific query, etc) An archive (tar or zip) of the dbpath/diagnostic.data directory If you'd prefer to keep any of this information private, we can generate a secure upload portal for you. One other thing I would note: as I mentioned previously, mongod needs to be able to perform fsync on directories - I would be curious to know if you can recreate this behavior when not using a shared drive for your dbpath Thanks, Nick kelvinl commented on Mon, 20 Aug 2018 23:34:53 +0000: Nick, I was out for vacation and did not reply your comment. I re-installed MongoDB with most recent version 4.01 and reload data. I do not see that bug again. However, I experienced following error from time to time with { driver: { name: "mongo-csharp-driver", version: "2.7.0.0" }, os: { type: "Windows", name: "Microsoft Windows 10.0.10240", architecture: "x86_64", version: "10.0.10240" }, platform: ".NET Core 4.6.26515.07" } 2018-08-20T22:11:03.841+0000 I NETWORK [conn228] received client metadata from 10.10.25.26:60642 conn228: { driver: { name: "mongo-csharp-driver", version: "2.7.0.0" }, os: { type: "Windows", name: "Microsoft Windows 10.0.10240", architecture: "x86_64", version: "10.0.10240" }, platform: ".NET Core 4.6.26515.07" } 2018-08-20T22:32:40.614+0000 F - [WTCheckpointThread] Invalid access at address: 0x4 2018-08-20T22:32:40.745+0000 F - [WTCheckpointThread] Got signal: 11 (Segmentation fault). 0x55fcfa781bf1 0x55fcfa780e09 0x55fcfa781476 0x7f342dec2890 0x55fcf8f3caa9 0x55fcf8f3dad6 0x55fcf8f42feb 0x55fcf8f9ef04 0x55fcf8f6f06a 0x55fcf8f714e6 0x55fcf8f72503 0x55fcf8f580ba 0x55fcf8ed0aef 0x55fcfa2c2771 0x55fcfa8910b0 0x7f342deb76db 0x7f342dbe088f ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"55FCF8404000","o":"237DBF1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55FCF8404000","o":"237CE09"},{"b":"55FCF8404000","o":"237D476"},{"b":"7F342DEB0000","o":"12890"},{"b":"55FCF8404000","o":"B38AA9"},{"b":"55FCF8404000","o":"B39AD6"},{"b":"55FCF8404000","o":"B3EFEB","s":"__wt_reconcile"},{"b":"55FCF8404000","o":"B9AF04","s":"__wt_cache_op"},{"b":"55FCF8404000","o":"B6B06A"},{"b":"55FCF8404000","o":"B6D4E6"},{"b":"55FCF8404000","o":"B6E503","s":"__wt_txn_checkpoint"},{"b":"55FCF8404000","o":"B540BA"},{"b":"55FCF8404000","o":"ACCAEF","s":"_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv"},{"b":"55FCF8404000","o":"1EBE771","s":"_ZN5mongo13BackgroundJob7jobBodyEv"},{"b":"55FCF8404000","o":"248D0B0"},{"b":"7F342DEB0000","o":"76DB"},{"b":"7F342DABF000","o":"12188F","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.1", "gitVersion" : "54f1582fc6eb01de4d4c42f26fc133e623f065fb", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "4.15.0-30-generic", "version" : "#32-Ubuntu SMP Thu Jul 26 17:42:43 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "55FCF8404000", "elfType" : 3, "buildId" : "277667F916B0411F06935B33EDB63802E236E5DA" }, { "b" : "7FFD1A4F7000", "path" : "linux-vdso.so.1", "elfType" : 3, "buildId" : "C65E0B5983446DCDF584BC5707F2CCBF32AA1884" }, { "b" : "7F342F38E000", "path" : "/usr/lib/x86_64-linux-gnu/libcurl.so.4", "elfType" : 3, "buildId" : "6AF728F07FFCDB4DAABE4F4CFC6EDB1401755DC7" }, { "b" : "7F342F173000", "path" : "/lib/x86_64-linux-gnu/libresolv.so.2", "elfType" : 3, "buildId" : "390E9CC4C215314B6D8ADE6D6E28F8518418039C" }, { "b" : "7F342ECFB000", "path" : "/usr/lib/x86_64-linux-gnu/libcrypto.so.1.1", "elfType" : 3, "buildId" : "DA6345B41E06A9001B5C51DEAF0784BE9D49810D" }, { "b" : "7F342EA91000", "path" : "/usr/lib/x86_64-linux-gnu/libssl.so.1.1", "elfType" : 3, "buildId" : "9787FF1FFFE8D82268D89F6BFA3AF07B3E9D140F" }, { "b" : "7F342E88D000", "path" : "/lib/x86_64-linux-gnu/libdl.so.2", "elfType" : 3, "buildId" : "25AD56E902E23B490A9CCDB08A9744D89CB95BCC" }, { "b" : "7F342E685000", "path" : "/lib/x86_64-linux-gnu/librt.so.1", "elfType" : 3, "buildId" : "9826FBDF57ED7D6965131074CB3C08B1009C1CD8" }, { "b" : "7F342E2E7000", "path" : "/lib/x86_64-linux-gnu/libm.so.6", "elfType" : 3, "buildId" : "A33761AB8FB485311B3C85BF4253099D7CABE653" }, { "b" : "7F342E0CF000", "path" : "/lib/x86_64-linux-gnu/libgcc_s.so.1", "elfType" : 3, "buildId" : "039AE85FEF075EC14FE3528762A0645C8CF73B29" }, { "b" : "7F342DEB0000", "path" : "/lib/x86_64-linux-gnu/libpthread.so.0", "elfType" : 3, "buildId" : "28C6AADE70B2D40D1F0F3D0A1A0CAD1AB816448F" }, { "b" : "7F342DABF000", "path" : "/lib/x86_64-linux-gnu/libc.so.6", "elfType" : 3, "buildId" : "B417C0BA7CC5CF06D1D1BED6652CEDB9253C60D0" }, { "b" : "7F342F60D000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "64DF1B961228382FE18684249ED800AB1DCEAAD4" }, { "b" : "7F342D89A000", "path" : "/usr/lib/x86_64-linux-gnu/libnghttp2.so.14", "elfType" : 3, "buildId" : "4F00E5207693FDC249DA42EC6472ACA6A7B929AE" }, { "b" : "7F342D67D000", "path" : "/usr/lib/x86_64-linux-gnu/libidn2.so.0", "elfType" : 3, "buildId" : "BA5BF9A5C44F48C647E9D8270A5421AE81CCAD61" }, { "b" : "7F342D461000", "path" : "/usr/lib/x86_64-linux-gnu/librtmp.so.1", "elfType" : 3, "buildId" : "69465D8AA6B19086ABF2455A703F9168BF82A69F" }, { "b" : "7F342D253000", "path" : "/usr/lib/x86_64-linux-gnu/libpsl.so.5", "elfType" : 3, "buildId" : "CDAF1F1946846941F9D06414EC8C812D131A168E" }, { "b" : "7F342D008000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "451E2EC02C8F974B21D1CE47F5D2C21A21C16A9F" }, { "b" : "7F342CDB6000", "path" : "/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2", "elfType" : 3, "buildId" : "0CDDF9D1F87FE7FF07D73DBE1D7CB57EE5BA802C" }, { "b" : "7F342CBA8000", "path" : "/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2", "elfType" : 3, "buildId" : "DD2893FC0AA3C58798F805D4C88EE086564D2EEB" }, { "b" : "7F342C98B000", "path" : "/lib/x86_64-linux-gnu/libz.so.1", "elfType" : 3, "buildId" : "EF3E006DFE3132A41D4D4DC0E407D6EA658E11C4" }, { "b" : "7F342C60D000", "path" : "/usr/lib/x86_64-linux-gnu/libunistring.so.2", "elfType" : 3, "buildId" : "62A44D226F3A74CF720305A52D3E876E214C248A" }, { "b" : "7F342C2A8000", "path" : "/usr/lib/x86_64-linux-gnu/libgnutls.so.30", "elfType" : 3, "buildId" : "F67AAC89381F6ACFE9979B24D4002668C4F70ABE" }, { "b" : "7F342C074000", "path" : "/usr/lib/x86_64-linux-gnu/libhogweed.so.4", "elfType" : 3, "buildId" : "842BDF0B0EAAB82E19F1EABFC38769F4040FBE31" }, { "b" : "7F342BE3E000", "path" : "/usr/lib/x86_64-linux-gnu/libnettle.so.6", "elfType" : 3, "buildId" : "C20D4B3BA13FCDCC3BF6857689BA9FC70BE3F6A5" }, { "b" : "7F342BBBD000", "path" : "/usr/lib/x86_64-linux-gnu/libgmp.so.10", "elfType" : 3, "buildId" : "D40EA9B5EC5BC46799E4A412319617BD38BE9341" }, { "b" : "7F342B8E7000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.3", "elfType" : 3, "buildId" : "AFF9A2A2B6ACB066BF450CC4A0772B9DC04299CB" }, { "b" : "7F342B6B5000", "path" : "/usr/lib/x86_64-linux-gnu/libk5crypto.so.3", "elfType" : 3, "buildId" : "04AADCB371B97A570756D69A0D5B3B0E6737E349" }, { "b" : "7F342B4B1000", "path" : "/lib/x86_64-linux-gnu/libcom_err.so.2", "elfType" : 3, "buildId" : "9E885FB7F54DA464A3D2A978C71C1178A6EC7512" }, { "b" : "7F342B2A6000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5support.so.0", "elfType" : 3, "buildId" : "27A5983855C50A37EFA526BB12EA48CDAE3B5FCA" }, { "b" : "7F342B08B000", "path" : "/usr/lib/x86_64-linux-gnu/libsasl2.so.2", "elfType" : 3, "buildId" : "ABB7E3F40302E6509DAD1F91DFB1F04B6A5FD072" }, { "b" : "7F342AE4A000", "path" : "/usr/lib/x86_64-linux-gnu/libgssapi.so.3", "elfType" : 3, "buildId" : "A1A98DB481968073636BBAECB561A3EA8ED198AE" }, { "b" : "7F342AB1B000", "path" : "/usr/lib/x86_64-linux-gnu/libp11-kit.so.0", "elfType" : 3, "buildId" : "8DBD451EA5651283905E16FA7DFA9908688893A3" }, { "b" : "7F342A908000", "path" : "/usr/lib/x86_64-linux-gnu/libtasn1.so.6", "elfType" : 3, "buildId" : "6036B89A3BB671B32E01464C0C82BFA016186352" }, { "b" : "7F342A704000", "path" : "/lib/x86_64-linux-gnu/libkeyutils.so.1", "elfType" : 3, "buildId" : "F463E107B099910463BC32E837C73D341A52C27B" }, { "b" : "7F342A4FB000", "path" : "/usr/lib/x86_64-linux-gnu/libheimntlm.so.0", "elfType" : 3, "buildId" : "C2376C5B831991591F1A67B976758185F86896D8" }, { "b" : "7F342A26E000", "path" : "/usr/lib/x86_64-linux-gnu/libkrb5.so.26", "elfType" : 3, "buildId" : "69BDEE5FA0FEEDF317308BE850F78761861D520A" }, { "b" : "7F3429FCC000", "path" : "/usr/lib/x86_64-linux-gnu/libasn1.so.8", "elfType" : 3, "buildId" : "315D74995AAA32DE4D15BA25F335066988B1B230" }, { "b" : "7F3429D96000", "path" : "/usr/lib/x86_64-linux-gnu/libhcrypto.so.4", "elfType" : 3, "buildId" : "6673972A1C24A89EBAFBAE696188A4CB26C6DDEB" }, { "b" : "7F3429B80000", "path" : "/usr/lib/x86_64-linux-gnu/libroken.so.18", "elfType" : 3, "buildId" : "430827C33259C12248CF44B91A9A9821114376F5" }, { "b" : "7F3429978000", "path" : "/usr/lib/x86_64-linux-gnu/libffi.so.6", "elfType" : 3, "buildId" : "3555B5F599C9787DFDDBF9E8DF6F706B9044D985" }, { "b" : "7F342974F000", "path" : "/usr/lib/x86_64-linux-gnu/libwind.so.0", "elfType" : 3, "buildId" : "93A0931B1C2818F0EA224CE6FE5E31E84A9B55BB" }, { "b" : "7F3429540000", "path" : "/usr/lib/x86_64-linux-gnu/libheimbase.so.1", "elfType" : 3, "buildId" : "669D4CCE42FA4382796EFFCF0C16F459F4382C4C" }, { "b" : "7F34292F6000", "path" : "/usr/lib/x86_64-linux-gnu/libhx509.so.5", "elfType" : 3, "buildId" : "4B80C543356EE0AF9039EFE7C9EA1CC1F74C426A" }, { "b" : "7F3428FED000", "path" : "/usr/lib/x86_64-linux-gnu/libsqlite3.so.0", "elfType" : 3, "buildId" : "9FD7B69263297314C83BB205234E182BD4D8B86A" }, { "b" : "7F3428DB5000", "path" : "/lib/x86_64-linux-gnu/libcrypt.so.1", "elfType" : 3, "buildId" : "810686AF0D5FD350A4FB1CC4B5AFF44A05C102CB" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55fcfa781bf1] mongod(+0x237CE09) [0x55fcfa780e09] mongod(+0x237D476) [0x55fcfa781476] libpthread.so.0(+0x12890) [0x7f342dec2890] mongod(+0xB38AA9) [0x55fcf8f3caa9] mongod(+0xB39AD6) [0x55fcf8f3dad6] mongod(__wt_reconcile+0x297B) [0x55fcf8f42feb] mongod(__wt_cache_op+0x594) [0x55fcf8f9ef04] mongod(+0xB6B06A) [0x55fcf8f6f06a] mongod(+0xB6D4E6) [0x55fcf8f714e6] mongod(__wt_txn_checkpoint+0x1C3) [0x55fcf8f72503] mongod(+0xB540BA) [0x55fcf8f580ba] mongod(_ZN5mongo18WiredTigerKVEngine26WiredTigerCheckpointThread3runEv+0x59F) [0x55fcf8ed0aef] mongod(_ZN5mongo13BackgroundJob7jobBodyEv+0x131) [0x55fcfa2c2771] mongod(+0x248D0B0) [0x55fcfa8910b0] libpthread.so.0(+0x76DB) [0x7f342deb76db] libc.so.6(clone+0x3F) [0x7f342dbe088f] ----- END BACKTRACE ----- nick.brewer commented on Mon, 20 Aug 2018 15:40:59 +0000: kelvinl Since we haven't heard back from you, I'm going to close this issue for now. One thing I'll note is that it's possible you were running into an fsync issue due to the use of the shared Virtualbox drive. -Nick nick.brewer commented on Wed, 1 Aug 2018 14:59:11 +0000: kelvinl Thanks for the info. We'll await your results. -Nick kelvinl commented on Tue, 31 Jul 2018 21:24:15 +0000: I use Ubuntu 18.04 on Oracle Virtual Box and dbpath point to /mnt is just another mount point from different logic drive file. I will use your latest binary to build a new mongoDB 4.0 Server with Ubuntu 18.04 and start over again and will let you know if I run to any issues later nick.brewer commented on Tue, 31 Jul 2018 20:54:14 +0000: kelvinl Sorry, this is the correct set of binaries to use for your testing. From your log, I noticed that the dbpath is pointing to a directory with /mnt - is this an external device? And if so, what kind? It would be helpful to get the device's corresponding entry in /etc/fstab, if it has one. Thanks, Nick nick.brewer commented on Tue, 31 Jul 2018 15:14:17 +0000: kelvinl MongoDB 4.0.0 is not officially supported on Ubuntu 18.04 - were you using a binary for an older release? An 18.04 binary is available for our current development release, which you can find here. Could you try recreating this on the latest version? Note that you'll need to test on a new database (starting over from a clean dbpath). Thanks, Nick
Go to http://openaddresses.io download address data and extract to folder as testing data. Use attached source code in src.zip to run in debug model for 4-5 hours to reproduce this issue. (Use MongoDB 2.7 C# Driver) Please change following in program.cs file to your local location const string ROOT_PATH = @"E:\temp\Address\us\tx"; const string ARCHIVE = @"E:\temp\Address\archive"; const string INDEX_PATH = @"E:\temp\Address\index";