Info

Hello We didn't get much clear resolution in SERVER-38292 so am opening a new ticket now that the server version has been upgraded to 4.0.14. The problem seem to have been resurfaced. Let me know what all additional information is required to be uploaded via your secure portal, and I'll do the needful. 2020-02-14T14:40:26.821-0600 F - [thread578349] Invalid access at address: 0 2020-02-14T14:40:26.849-0600 F - [thread578349] Got signal: 11 (Segmentation fault). 0x56294090acc1 0x562940909ed9 0x56294090a546 0x7efea7caf5d0 0x5629409f7d03 0x5629409f7ddc 0x562940a9a5da 0x56293f03eb9d 0x56293f03efa2 0x56293f083288 0x56293f085428 0x56293f088ceb 0x56293efff4c4 0x56293f000bd2 0x56293eff95b4 0x56293eff9b23 0x56293effbe77 0x56293f04a569 0x7efea7ca7dd5 0x7efea79d0ead ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"56293E4AA000","o":"2460CC1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"56293E4AA000","o":"245FED9"},{"b":"56293E4AA000","o":"2460546"},{"b":"7EFEA7CA0000","o":"F5D0"},{"b":"56293E4AA000","o":"254DD03","s":"_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi"},{"b":"56293E4AA000","o":"254DDDC","s":"_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm"},{"b":"56293E4AA000","o":"25F05DA","s":"_ZdlPvRKSt9nothrow_t"},{"b":"56293E4AA000","o":"B94B9D"},{"b":"56293E4AA000","o":"B94FA2","s":"__wt_stash_add"},{"b":"56293E4AA000","o":"BD9288"},{"b":"56293E4AA000","o":"BDB428"},{"b":"56293E4AA000","o":"BDECEB","s":"__wt_split_reverse"},{"b":"56293E4AA000","o":"B554C4"},{"b":"56293E4AA000","o":"B56BD2","s":"__wt_evict"},{"b":"56293E4AA000","o":"B4F5B4"},{"b":"56293E4AA000","o":"B4FB23"},{"b":"56293E4AA000","o":"B51E77","s":"__wt_evict_thread_run"},{"b":"56293E4AA000","o":"BA0569"},{"b":"7EFEA7CA0000","o":"7DD5"},{"b":"7EFEA78D3000","o":"FDEAD","s":"clone"}],"processInfo":{ "mongodbVersion" : "4.0.14", "gitVersion" : "1622021384533dade8b3c89ed3ecd80e1142c132", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-957.1.3.el7.x86_64", "version" : "#1 SMP Thu Nov 29 14:49:43 UTC 2018", "machine" : "x86_64" }, "somap" : [ { "b" : "56293E4AA000", "elfType" : 3, "buildId" : "198875DDA28BA207732ADCC6DB541F81EE7988EF" }, { "b" : "7FFD5AA48000", "elfType" : 3, "buildId" : "DF8F6BF69E976BF1266E476EA2E37CEE06F10C1D" }, { "b" : "7EFEA90CC000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "89C83CEB5DE5FDEC4F3DFBA4FCAACB53D747A998" }, { "b" : "7EFEA8EB3000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "4C488F6E7044BB966162C1F7081ABBA6EBB2B485" }, { "b" : "7EFEA8A52000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "8BD89856B64DD5189BF075EF574EDF203F93D44A" }, { "b" : "7EFEA87E0000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "AEF5E6F2240B55F90E9DF76CFBB8B9D9F5286583" }, { "b" : "7EFEA85DC000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "67AD3498AC7DE3EAB952A243094DF5C12A21CD7D" }, { "b" : "7EFEA83D4000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "EFDE2029C9A4A20BE5B8D8AE7E6551FF9B5755D2" }, { "b" : "7EFEA80D2000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "918D3696BF321AA8D32950AB2AB8D0F1B21AC907" }, { "b" : "7EFEA7EBC000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "6B4F3D896CD0F06FCB3DEF0245F204ECE3220D7E" }, { "b" : "7EFEA7CA0000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "3D9441083D079DC2977F1BD50C8068D11767232D" }, { "b" : "7EFEA78D3000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "3C61131D1DAC9DA79B73188E7702BEF786C2AD54" }, { "b" : "7EFEA9336000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "5DA2D47925497B2F5875A7D8D1799A1227E2FDE4" }, { "b" : "7EFEA76A0000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7EFEA7473000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "1AF123CADB2F2910E89CBD540A06D3B33692F95E" }, { "b" : "7EFEA7221000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "DE2FC96025F4F43AA6E48523D354581D2C5E0669" }, { "b" : "7EFEA6FFA000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "9757EB6D93E31024C436919A89BEF33CA4F1B984" }, { "b" : "7EFEA6CCD000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "289D245DC413161931C170A1DFB9676582C84DBF" }, { "b" : "7EFEA6A9E000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "B7F45CBBFB31A1390DA3CD65494FB4824046B22C" }, { "b" : "7EFEA689A000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "084D2194302908913F68B9DCD27DE46FA5B50522" }, { "b" : "7EFEA6695000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "799B28AD9A5460D78376E2C11260F2E858B95DE3" }, { "b" : "7EFEA6457000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "DE762A28174110911B273E175D54F222B313CFE0" }, { "b" : "7EFEA620A000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "B5C83BDE7ED7026835B779FA0F957FCCCD599F40" }, { "b" : "7EFEA5F21000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "8B63976509135BA73A12153D6FDF7B3B9E5D2A54" }, { "b" : "7EFEA5D06000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "6183129B5F29CA14580E517DF94EF317761FA6C9" }, { "b" : "7EFEA5B02000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "B4BE1023D9606A88169DF411BF94AF417D7BA1A0" }, { "b" : "7EFEA58F3000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "3192C56CD451E18EB9F29CB045432BA9C738DD29" }, { "b" : "7EFEA569E000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "B0F2F559615F60699047274ABA36A9FAA1FE7C06" }, { "b" : "7EFEA5488000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B9D5F73428BD6AD68C96986B57BEA3B7CEDB9745" }, { "b" : "7EFEA5279000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "98F619035053EF68358099CE7CF1AA528B3B229D" }, { "b" : "7EFEA5075000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7EFEA4E58000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7EFEA4C31000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "D2DD4DA3FDE1477D25BFFF80F3A25FDB541A8179" }, { "b" : "7EFEA49FA000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "740CAD898E29E1F3B73A323CCEC4A7C88911647F" }, { "b" : "7EFEA4798000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7EFEA4595000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "B758881F4B6AF6C28C07A1A57713CBD2144628D4" } ] }} mongod-4.0(_ZN5mongo15printStackTraceERSo+0x41) [0x56294090acc1] mongod-4.0(+0x245FED9) [0x562940909ed9] mongod-4.0(+0x2460546) [0x56294090a546] libpthread.so.0(+0xF5D0) [0x7efea7caf5d0] mongod-4.0(_ZN8tcmalloc11ThreadCache21ReleaseToCentralCacheEPNS0_8FreeListEmi+0xE3) [0x5629409f7d03] mongod-4.0(_ZN8tcmalloc11ThreadCache11ListTooLongEPNS0_8FreeListEm+0x1C) [0x5629409f7ddc] mongod-4.0(_ZdlPvRKSt9nothrow_t+0x26A) [0x562940a9a5da] mongod-4.0(+0xB94B9D) [0x56293f03eb9d] mongod-4.0(__wt_stash_add+0xA2) [0x56293f03efa2] mongod-4.0(+0xBD9288) [0x56293f083288] mongod-4.0(+0xBDB428) [0x56293f085428] mongod-4.0(__wt_split_reverse+0x11B) [0x56293f088ceb] mongod-4.0(+0xB554C4) [0x56293efff4c4] mongod-4.0(__wt_evict+0x1682) [0x56293f000bd2] mongod-4.0(+0xB4F5B4) [0x56293eff95b4] mongod-4.0(+0xB4FB23) [0x56293eff9b23] mongod-4.0(__wt_evict_thread_run+0x77) [0x56293effbe77] mongod-4.0(+0xBA0569) [0x56293f04a569] libpthread.so.0(+0x7DD5) [0x7efea7ca7dd5] libc.so.6(clone+0x6D) [0x7efea79d0ead] ----- END BACKTRACE -----

Top User Comments

syseng.mahesh@gmail.com commented on Mon, 28 Mar 2022 12:07:40 +0000: @Manan or @Sanjeeth do you have a fix or workaround for this issue. JIRAUSER1262199 commented on Thu, 2 Sep 2021 12:57:14 +0000: We're encountering same error on `4.2.12` & `4.2.15` version Mongod. Signal 11 is raised and mongo service entering failed/crashed state. Segementation Fault is observed on all other related nodes having same mongod `4.2.12` & `4.2.15` 2021-08-13T15:30:01.237+0000 F  - [conn684] Invalid access at address: 0x55650b02a0002021-08-13T15:30:01.285+0000 F  - [conn684] Got signal: 11 (Segmentation fault). 0x55626cc65241 0x55626cc6477e 0x55626cc6495c 0x7f63495c1630 0x7f634535c41c----- BEGIN BACKTRACE -----{"backtrace":[{"b":"55626A306000","o":"295F241","s":"_ZN5mongo15printStackTraceERSo"},{"b":"55626A306000","o":"295E77E"},{"b":"55626A306000","o":"295E95C"},{"b":"7F63495B2000","o":"F630"},{"b":"7F6345310000","o":"4C41C"}],"processInfo":{ "mongodbVersion" : "4.2.15", "gitVersion" : "d7fd78dead621a539c20791a93abec34bb1be385", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1160.36.2.el7.x86_64", "version" : "#1 SMP Wed Jul 21 11:57:15 UTC 2021", "machine" : "x86_64" }, "somap" : [ { "b" : "55626A306000", "elfType" : 3, "buildId" : "3B94268B834009D9D3B84B8E1A69A6F240DD54C5" }, { "b" : "7FFEE3C8E000", "elfType" : 3, "buildId" : "1D0D0FA3882A1742CF8DE0974C7FC6FC7CAD8185" }, { "b" : "7F634A93F000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "AA185EEE9B8EE0BCE652F7A6E9AAD07B1B3808C4" }, { "b" : "7F634A725000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "B45C711D26DDD9F612D7814CE83B427927C8BC65" }, { "b" : "7F634A365000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "7829C3727DE1DD529EBA8E477756D10267F5656E" }, { "b" : "7F634A0F2000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "4474A303DA0EA2DF580F0648779A5E8F62F0433A" }, { "b" : "7F6349EEE000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7F2E9CB0769D7E57BD669B485A74B537B63A57C4" }, { "b" : "7F6349CE6000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "3E44DF7055942478D052E40FDD1F5B7862B152B0" }, { "b" : "7F63499E4000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7615604EAF4A068DFAE5085444D15C0DEE93DFBD" }, { "b" : "7F63497CE000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDF51350C7F71496149D064AA8B1441F786DF88A" }, { "b" : "7F63495B2000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B" }, { "b" : "7F63491E4000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A317B42B15368ADCAE21C11107691A03EC91059D" }, { "b" : "7F634ABA9000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "62C449974331341BB08DCCE3859560A22AF1E172" }, { "b" : "7F6348FB1000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "2B77BBEFFF65E94F3E0B71A4E89BEB68C4B476C5" }, { "b" : "7F6348D84000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "CB0BD6C014F41EC926FAC41322C82FF4A5EB88B9" }, { "b" : "7F6348B27000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "3B88957E9F14880EA8331F24EADD1B48C37FF032" }, { "b" : "7F63488FF000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "CED43363B6A38A426D2A5EEECC7A267DD7BFBD60" }, { "b" : "7F63485CA000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "3A70799DB1310CAFAE52B163B3046EBA6C7AA10D" }, { "b" : "7F634839A000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "A61604F9C4E3F975A0A1742174F08D6ECF987A63" }, { "b" : "7F6348196000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "E761D8698407F6521F26F579D61D5EC8F7EF04A9" }, { "b" : "7F6347F91000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "41E234507D6BB1E4FE56A078127D36C1963460CC" }, { "b" : "7F6347D53000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "051956498509E67F79215B76781C1AA0504EB5D5" }, { "b" : "7F6347B06000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "0CAEC124D97114DA40DDEB0FED1FAD5D14C3D626" }, { "b" : "7F634781D000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "4EBF28968DA5784ED6606BFF7C1915C50AC24502" }, { "b" : "7F63475EA000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "5FF9D1075A8D5D62F77F5CE56C935FCD92C62EFA" }, { "b" : "7F63473E6000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "2ADDB65846A50CE45F0C9B62EA35DDA62C6AD7A2" }, { "b" : "7F63471D7000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "8804516A3226CFB54589FEC0E27D89C93DAF92FF" }, { "b" : "7F6346F82000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "82277E588B8B663A0A53F4E9516AEC1802BD4FC3" }, { "b" : "7F6346D6C000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "B04855870B0DE434F354DE3147230F2677200B56" }, { "b" : "7F6346B04000", "path" : "/lib64/libbwrap.so.1", "elfType" : 3, "buildId" : "0B5E54BCDB9970EB56C5F8A7029969ABB6C37B1F" }, { "b" : "7F63468F4000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "779381063DAECC27E8480C8F79F0651162586478" }, { "b" : "7F63466F0000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "2E01D5AC08C1280D013AAB96B292AC58BC30A263" }, { "b" : "7F63464D3000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "E2F2017F821DD1B9D307DA1A9B8014F2941AEB7B" }, { "b" : "7F63462AC000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "903A0BD0BFB4FEE8C284F41BEB9773DED94CBC52" }, { "b" : "7F6346075000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "97BE6F9199FED4491B00AA91F7E6EACC4D5328F7" }, { "b" : "7F6345E13000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "9CA3D11F018BEEB719CDB34BE800BF1641350D0A" }, { "b" : "7F6345C10000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "020C788B41DCC71AEE66B822D7670BC4347DA006" }, { "b" : "7F63459C2000", "path" : "/usr/lib64/bwrap-1.5/libcryptocme.so", "elfType" : 3, "buildId" : "2DD77A93881EE4CFB883B14BC6D8D10C3184B38C" }, { "b" : "7F63457BC000", "path" : "/usr/lib64/bwrap-1.5/libccme_error_info.so", "elfType" : 3, "buildId" : "DDFAB5F4AB36FD6F8F3D63D980D448871940F310" }, { "b" : "7F63455B6000", "path" : "/usr/lib64/bwrap-1.5/libccme_aux_entropy.so", "elfType" : 3, "buildId" : "72880238B13986785D452F3319EDE010806338E6" }, { "b" : "7F6345310000", "path" : "/usr/lib64/bwrap-1.5/libccme_base.so", "elfType" : 3, "buildId" : "A9269B4C5DAB2A2C323871125515D887340EB64A" }, { "b" : "7F634508D000", "path" : "/usr/lib64/bwrap-1.5/libccme_asym.so", "elfType" : 3, "buildId" : "4F591A87B4AA7D6D7A019BE035C0DAEF19CBF3B2" }, { "b" : "7F6344D99000", "path" : "/usr/lib64/bwrap-1.5/libccme_ecc.so", "elfType" : 3, "buildId" : "C9D87694F8B506588CE5BB4019EEEF78485E09A8" }, { "b" : "7F6344B05000", "path" : "/usr/lib64/bwrap-1.5/libccme_base_non_fips.so", "elfType" : 3, "buildId" : "949E6E036E8FA3697242B222D379FACC58102A0B" }, { "b" : "7F63448E6000", "path" : "/usr/lib64/bwrap-1.5/libccme_ecc_non_fips.so", "elfType" : 3, "buildId" : "B622529118F685678E12F2858980DCE223012434" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x55626cc65241] mongod(+0x295E77E) [0x55626cc6477e] mongod(+0x295E95C) [0x55626cc6495c] libpthread.so.0(+0xF630) [0x7f63495c1630] libccme_base.so(+0x4C41C) [0x7f634535c41c]-----  END BACKTRACE  ----- manan@indeed.com commented on Fri, 21 Aug 2020 17:25:09 +0000: @alexander / @kelsey Can I know if you got a chance to look thru the diagnostic files I uploaded previously here? The issue is still plaguing once every 2-3 weeks on a few specific hosts of this replica set. I just want to know with evidence if this is peculiar to the hardware issue on these hosts or something else? For some reason, it hasn't occurred on three other replicas of this same set that are in two another data centers. The only difference is the read traffic is extremely light in those two data center replicas (1/1000th qps reads). manan@indeed.com commented on Sun, 26 Apr 2020 17:04:12 +0000: While this does not happen every day, it is still a problem in production. This happened after 20 days again today. I hope you can get me some information from the diagnostic data files. Do you think re-syncing data from scratch can mitigate the problem? manan@indeed.com commented on Mon, 6 Apr 2020 18:43:17 +0000: Alexander, this is still happening. Can you please provide us a clue or a work around? Is that anything found from the diagnostic data or other files I uploaded on Feb 16? manan@indeed.com commented on Fri, 21 Feb 2020 19:26:25 +0000: I verified we are not using $sample in any of the application code. alexander.gorrod commented on Fri, 21 Feb 2020 02:48:14 +0000: manan@indeed.com I wonder - do you use $sample in your application? There is one known failure that could be related to this, but it's specific to cases where $sample (which use WiredTiger random cursors underneath) is used. manan@indeed.com commented on Sun, 16 Feb 2020 21:32:39 +0000: Hi there, I've uploaded the requested files on the upload portal directly via curl. Please let me know if you notice anything is missing. thomas.schubert commented on Fri, 14 Feb 2020 22:08:20 +0000: Hi manan@indeed.com Thanks for opening a new ticket. To continue to investigate, would you please provide the following: The mongod.log file The content of the diagnostic.data directory Syslog messages The results of running "ls -l" on the database home directory I've created an upload portal for you to use here. Thank again, Kelsey manan@indeed.com commented on Fri, 14 Feb 2020 21:01:23 +0000: cc @alexander Gorrod

Steps to Reproduce