OPERATIONAL DEFECT DATABASE
...
...
BugZero found this defect 2733 days ago.
MongoDB server crashed because of serverStatus command of FTDC. And gitVersion might be different from official distribution, because I built mongod with random git version. (And I never changed wiredtiger section of serverStatus command and also FTDC) 2017-06-18T04:14:49.000+0900 F - [ftdc] Invalid access at address: 0 2017-06-18T04:14:49.012+0900 F - [ftdc] Got signal: 11 (Segmentation fault). 0x7ff494a9a5ca 0x7ff494a99849 0x7ff494a99e8f 0x7ff492548370 0x7ff4955093e6 0x7ff4921bf231 0x7ff4922848d5 0x7ff4954593f3 0x7ff495493bea 0x7ff49541e08b 0x7ff4947c1bd6 0x7ff4947b6423 0x7ff4940044da 0x7ff4940055b6 0x7ff4940f8c58 0x7ff4940eb85b 0x7ff4940ef9ab 0x7ff4955058df 0x7ff492540dc5 0x7ff49226f73d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"7FF4935A4000","o":"14F65CA","s":"_ZN5mongo15printStackTraceERSo"},{"b":"7FF4935A4000","o":"14F5849"},{"b":"7FF4935A4000","o":"14F5E8F"},{"b":"7FF492539000","o":"F370"},{"b":"7FF4935A4000","o":"1F653E6","s":"_ZdlPvRKSt9nothrow_t"},{"b":"7FF492178000","o":"47231","s":"_IO_vfprintf"},{"b":"7FF492178000","o":"10C8D5","s":"__vsnprintf_chk"},{"b":"7FF4935A4000","o":"1EB53F3","s":"__wt_vsnprintf_len_incr"},{"b":"7FF4935A4000","o":"1EEFBEA"},{"b":"7FF4935A4000","o":"1E7A08B"},{"b":"7FF4935A4000","o":"121DBD6","s":"_ZN5mongo14WiredTigerUtil17exportTableToBSONEP12__wt_sessionRKSsS4_PNS_14BSONObjBuilderE"},{"b":"7FF4935A4000","o":"1212423","s":"_ZNK5mongo29WiredTigerServerStatusSection15generateSectionEPNS_16OperationContextERKNS_11BSONElementE"},{"b":"7FF4935A4000","o":"A604DA","s":"_ZNK5mongo19ServerStatusSection13appendSectionEPNS_16OperationContextERKNS_11BSONElementEPNS_14BSONObjBuilderE"},{"b":"7FF4935A4000","o":"A615B6","s":"_ZN5mongo15CmdServerStatus3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE"},{"b":"7FF4935A4000","o":"B54C58"},{"b":"7FF4935A4000","o":"B4785B","s":"_ZN5mongo23FTDCCollectorCollection7collectEPNS_6ClientE"},{"b":"7FF4935A4000","o":"B4B9AB","s":"_ZN5mongo14FTDCController6doLoopEv"},{"b":"7FF4935A4000","o":"1F618DF"},{"b":"7FF492539000","o":"7DC5"},{"b":"7FF492178000","o":"F773D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.4", "gitVersion" : "888390515874a9debd1b6c5d36559ca86b44babd", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-514.16.1.el7.x86_64", "version" : "#1 SMP Wed Apr 12 15:04:24 UTC 2017", "machine" : "x86_64" }, "somap" : [ { "b" : "7FF4935A4000", "elfType" : 3, "buildId" : "ACE65D1BE5A49573AC50D3472F8CD399B34E9E1F" }, { "b" : "7FFF612E1000", "elfType" : 3, "buildId" : "B1A473E77A06C30ABAADD55464231F0FCFA9D0BF" }, { "b" : "7FF49317A000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "82E77ADE22BC9FFF8D3458BD37331E7EDF174C28" }, { "b" : "7FF492F76000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "C5F560504E1AF52E29679C3B52FF11121015D6BB" }, { "b" : "7FF492C6D000", "path" : "/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "913C54197A79038476BCA584355952C7C3B719F9" }, { "b" : "7FF49296B000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "721C7CC9488EFA25F83B48AF713AB27DBE48EF3E" }, { "b" : "7FF492755000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "408B46E291B2D4C9612E27C0509D165D7E186D40" }, { "b" : "7FF492539000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "C3DEB1FA27CD0C1C3CC575B944ABACBA0698B0F2" }, { "b" : "7FF492178000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8B2C421716985B927AA0CAF2A05D0B1F452367F7" }, { "b" : "7FF493382000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "8F3E366E2DB73C330A3791DEAE31AE9579099B44" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x3A) [0x7ff494a9a5ca] mongod(+0x14F5849) [0x7ff494a99849] mongod(+0x14F5E8F) [0x7ff494a99e8f] libpthread.so.0(+0xF370) [0x7ff492548370] mongod(_ZdlPvRKSt9nothrow_t+0x86) [0x7ff4955093e6] libc.so.6(_IO_vfprintf+0x101) [0x7ff4921bf231] libc.so.6(__vsnprintf_chk+0x95) [0x7ff4922848d5] mongod(__wt_vsnprintf_len_incr+0x23) [0x7ff4954593f3] mongod(+0x1EEFBEA) [0x7ff495493bea] mongod(+0x1E7A08B) [0x7ff49541e08b] mongod(_ZN5mongo14WiredTigerUtil17exportTableToBSONEP12__wt_sessionRKSsS4_PNS_14BSONObjBuilderE+0x246) [0x7ff4947c1bd6] mongod(_ZNK5mongo29WiredTigerServerStatusSection15generateSectionEPNS_16OperationContextERKNS_11BSONElementE+0x183) [0x7ff4947b6423] mongod(_ZNK5mongo19ServerStatusSection13appendSectionEPNS_16OperationContextERKNS_11BSONElementEPNS_14BSONObjBuilderE+0x3A) [0x7ff4940044da] mongod(_ZN5mongo15CmdServerStatus3runEPNS_16OperationContextERKSsRNS_7BSONObjEiRSsRNS_14BSONObjBuilderE+0xEE6) [0x7ff4940055b6] mongod(+0xB54C58) [0x7ff4940f8c58] mongod(_ZN5mongo23FTDCCollectorCollection7collectEPNS_6ClientE+0x3CB) [0x7ff4940eb85b] mongod(_ZN5mongo14FTDCController6doLoopEv+0x24B) [0x7ff4940ef9ab] mongod(+0x1F618DF) [0x7ff4955058df] libpthread.so.0(+0x7DC5) [0x7ff492540dc5] libc.so.6(clone+0x6D) [0x7ff49226f73d] ----- END BACKTRACE -----
sunguck.lee@gmail.com commented on Mon, 26 Jun 2017 02:01:22 +0000: Hi Thomas. There's no issue after changing the server machine. You may close this issue (And also SERVER-29790). Thanks. thomas.schubert commented on Fri, 23 Jun 2017 13:27:43 +0000: Thank you Matt. That aligns with our suspicion after reviewing the stacks, and we'll wait for your update. sunguck.lee@gmail.com commented on Fri, 23 Jun 2017 05:41:52 +0000: Hi Thomas. We suffered 3 times mongod crash (on different thread stack) on the same server machine. So I am can't help being doubtful of hardware or somewhere of linux os. First of all, we will change the memory of the machine, and watching this issue for a while. I will comment in a few days again. Regards, Matt. sunguck.lee@gmail.com commented on Thu, 22 Jun 2017 01:44:41 +0000: More detailed stack trace. (gdb) bt #0 get (this=, k=0) at src/third_party/gperftools-2.5/src/pagemap.h:253 #1 GetDescriptor (p=0, this=) at src/third_party/gperftools-2.5/src/page_heap.h:137 #2 do_free_helper (invalid_free_fn=0x7ff494c603f0 , size_hint=0, use_hint=false, heap_must_be_valid=true, heap=0x7ff496af7780, ptr=0x0) at src/third_party/gperftools-2.5/src/tcmalloc.cc:1309 #3 do_free_with_callback (invalid_free_fn=0x7ff494c603f0 , size_hint=0, use_hint=false, ptr=0x0) at src/third_party/gperftools-2.5/src/tcmalloc.cc:1362 #4 do_free (ptr=0x0) at src/third_party/gperftools-2.5/src/tcmalloc.cc:1370 #5 tc_free (ptr=0x0) at src/third_party/gperftools-2.5/src/tcmalloc.cc:1635 #6 0x00007ff4921bf231 in vfprintf () from /lib64/libc.so.6 #7 0x00007ff4922848d5 in __vsnprintf_chk () from /lib64/libc.so.6 #8 0x00007ff4954593f3 in vsnprintf (__ap=__ap@entry=0x7ff477b0efb0, __fmt=__fmt@entry=0x7ff49550be42 "%lu", __n=, __s=) at /usr/include/bits/stdio2.h:77 #9 __wt_vsnprintf_len_incr (buf=, size=, retsizep=retsizep@entry=0x7ff477b0efa8, fmt=fmt@entry=0x7ff49550be42 "%lu", ap=ap@entry=0x7ff477b0efb0) at src/third_party/wiredtiger/src/os_posix/os_snprintf.c:22 #10 0x00007ff495493bea in __wt_vsnprintf_len_set (ap=0x7ff477b0efb0, fmt=0x7ff49550be42 "%lu", retsizep=0x7ff477b0efa8, size=, buf=) at src/third_party/wiredtiger/src/include/misc.i:160 #11 __wt_buf_fmt (session=0x7ff497625d40, buf=0x7ff49aee1328, fmt=0x7ff49550be42 "%lu") at src/third_party/wiredtiger/src/support/scratch.c:78 #12 0x00007ff49541e08b in __curstat_next (cursor=0x7ff49aee0a00) at src/third_party/wiredtiger/src/cursor/cur_stat.c:203 #13 0x00007ff4947c1bd6 in mongo::WiredTigerUtil::exportTableToBSON (session=session@entry=0x7ff497625d40, uri="statistics:", config="statistics=(fast)", bob=bob@entry=0x7ff477b0f2b0) at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp:483 #14 0x00007ff4947b6423 in mongo::WiredTigerServerStatusSection::generateSection (this=, txn=, configElement=...) at src/mongo/db/storage/wiredtiger/wiredtiger_server_status.cpp:70 #15 0x00007ff4940044da in mongo::ServerStatusSection::appendSection (this=0x7ff497598da0, txn=, configElement=..., result=0x7ff477b0f790) at src/mongo/db/commands/server_status.h:100 #16 0x00007ff4940055b6 in mongo::CmdServerStatus::run (this=, txn=, dbname=..., cmdObj=..., errmsg=..., result=...) at src/mongo/db/commands/server_status.cpp:138 #17 0x00007ff4940f8c58 in mongo::(anonymous namespace)::FTDCSimpleInternalCommandCollector::collect (this=, txn=, builder=...) at src/mongo/db/ftdc/ftdc_mongod.cpp:264 #18 0x00007ff4940eb85b in mongo::FTDCCollectorCollection::collect (this=this@entry=0x7ff4974a96f8, client=client@entry=0x7ff49ae9bd20) at src/mongo/db/ftdc/collector.cpp:88 #19 0x00007ff4940ef9ab in mongo::FTDCController::doLoop (this=0x7ff4974a9640) at src/mongo/db/ftdc/controller.cpp:215 #20 0x00007ff4955058df in execute_native_thread_routine () #21 0x00007ff492540dc5 in start_thread () from /lib64/libpthread.so.0 #22 0x00007ff49226f73d in clone () from /lib64/libc.so.6 sunguck.lee@gmail.com commented on Wed, 21 Jun 2017 03:21:53 +0000: Hi Thomas, I have uploaded mongod logs and diagnostic.data. And system information is ... # uname -a Linux hostname 3.10.0-514.16.1.el7.x86_64 #1 SMP Wed Apr 12 15:04:24 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Thank you. thomas.schubert commented on Tue, 20 Jun 2017 17:52:37 +0000: Hi matt.lee, Thanks for reporting this issue. So we can investigate, would you please build the debugsymbols and upload them to this portal? The complete mongod logs, syslogs, and diagnostic.data may also help us better understand what happened here, and can be uploaded to the same secure portal. Thank you, Thomas
I don't know how to reproduce this crash case.
