Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Over the last few days the python driver tests have been failing against master-nightly in Jenkins when run against a sharded cluster. It's not clear what test is tickling the bug, and seems to vary from test run to test run. I'm attaching the mongos log from one run. The backtrace is: ... 2015-06-17T00:18:43.068+0000 D SHARDING [conn46] found 0 collections left and 0 collections dropped for database pymongo_test_mike 2015-06-17T00:18:43.390+0000 D NETWORK [conn46] polling for status of connection to 127.0.0.1:27016, no events 2015-06-17T00:18:43.390+0000 D NETWORK [conn46] polling for status of connection to 127.0.0.1:27021, no events 2015-06-17T00:18:43.390+0000 D NETWORK [conn46] polling for status of connection to 127.0.0.1:27022, no events 2015-06-17T00:18:43.390+0000 D SHARDING [conn46] found 1 shards listed on config server(s) 2015-06-17T00:18:43.391+0000 F - [conn46] Invalid access at address: 0 2015-06-17T00:18:43.393+0000 F - [conn46] Got signal: 11 (Segmentation fault). 0xa98bb6 0xa982d2 0xa9860e 0x7f18029795b0 0x97122b 0x971d1b 0x99b237 0x9deff8 0x9dfbbc 0x9e57c8 0x9de624 0x600ace 0xa518c1 0x7f1802971f18 0x7f18026a7b2d ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"698BB6"},{"b":"400000","o":"6982D2"},{"b":"400000","o":"69860E"},{"b":"7F180296A000","o":"F5B0"},{"b":"400000","o":"57122B"},{"b":"400000","o":"571D1B"},{"b":"400000","o":"59B237"},{"b":"400000","o":"5DEFF8"},{"b":"400000","o":"5DFBBC"},{"b":"400000","o":"5E57C8"},{"b":"400000","o":"5DE624"},{"b":"400000","o":"200ACE"},{"b":"400000","o":"6518C1"},{"b":"7F180296A000","o":"7F18"},{"b":"7F18025C5000","o":"E2B2D"}],"processInfo":{ "mongodbVersion" : "3.1.5-pre-", "gitVersion" : "be88e42ef819aeb76b8625e646adf67626844c25", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.14.35-28.38.amzn1.x86_64", "version" : "#1 SMP Wed Mar 11 22:50:37 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "E68A888A295E8469F953C4C275B4EEC790FB1EB3" }, { "b" : "7FFE7669A000", "elfType" : 3, "buildId" : "8DD7A7F645E37D599573A937DBAA3E931F55DE40" }, { "b" : "7F180403E000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "354560FFC93703E5A80EEC8C66DF9E59DA335001" }, { "b" : "7F1803DFB000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "72C1DB5E2447A90D1BF34065BCC031B7263FFBAC" }, { "b" : "7F1803B8E000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "A5C6FCE56D21542B9535D5FBCFE74EBA71D24613" }, { "b" : "7F18037AA000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "75565934551BECB21A027BE0A3AE28A62AEEB7EE" }, { "b" : "7F18035A2000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "CC3700E5EACA3A520D1FD1BA3B469F2EFB1B8F11" }, { "b" : "7F180339E000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "3D5EB706F159BCCB9A3B3E5F528B512E2C500F90" }, { "b" : "7F180309A000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7F1802D9C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "226E54A756FDC288551092B4FDE818C8D6D6FC04" }, { "b" : "7F1802B86000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7F180296A000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "A8E8015F84374807F6920DF95D16BBAAAC02AB41" }, { "b" : "7F18025C5000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "D40AD0538A2FFAB96E5540EF73816C675CC0AF5E" }, { "b" : "7F1804259000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "37F70DE2ABEA184F24518EB5CC538FF0F54E081F" }, { "b" : "7F18023AE000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "F01971089728188EFA4AA473D8B2DE22CA6DE963" }, { "b" : "7F1802178000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "DD436290CC2AC5B442DFCEB885AA43C77D2C4CC7" }, { "b" : "7F1801E93000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "2B8787E8E0C317CF820E5D830D923BC744E497F4" }, { "b" : "7F1801C68000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B10FBFEC246C4EAD1719D16090D0BE54904BBFC9" }, { "b" : "7F1801A65000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "622F315EB5CB2F791E9B64020692EBA98195D06D" }, { "b" : "7F180185A000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "7292C0673D7C116E3389D3FFA67087A6B9287A71" }, { "b" : "7F1801657000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F1801441000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "87B4EBF2183C8EA4AB657212203EFFE6340E2F4F" }, { "b" : "7F18011C6000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "3A7566B3366A1989E66B5AD54CEAF58C57E2ED3A" }, { "b" : "7F1800FA5000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "803D7EF21A989677D056E52BAEB9AB5B154FB9D9" }, { "b" : "7F17FA878000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "6B82ECC04F262F87646E106FD82254D7B25684A6" }, { "b" : "7F17FA509000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "E3E19783ECE59CE43F5C2EAB3F2529DFE798EB3B" }, { "b" : "7F17FA304000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "916D3F3990557E30CB579ED3D144FA57774B9B6C" }, { "b" : "7F17FA0FF000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "7660C66EFF41AB59B1740F97C3E6F8D79D23567C" }, { "b" : "7F17F9EFA000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "523C058C8D946374FF6651FE748617BAC2A37371" } ] }} mongos(_ZN5mongo15printStackTraceERSo+0x26) [0xa98bb6] mongos(+0x6982D2) [0xa982d2] mongos(+0x69860E) [0xa9860e] libpthread.so.0(+0xF5B0) [0x7f18029795b0] mongos(_ZNK5mongo5Shard10runCommandERKSsRKNS_7BSONObjERS3_+0x5B) [0x97122b] mongos(_ZNK5mongo5Shard10runCommandERKSsS2_RNS_7BSONObjE+0x15B) [0x971d1b] mongos(+0x59B237) [0x99b237] mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x4B8) [0x9deff8] mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x26C) [0x9dfbbc] mongos(_ZN5mongo8Strategy15clientCommandOpERNS_7RequestE+0x1B8) [0x9e57c8] mongos(_ZN5mongo7Request7processEi+0x624) [0x9de624] mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x3E) [0x600ace] mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x261) [0xa518c1] libpthread.so.0(+0x7F18) [0x7f1802971f18] libc.so.6(clone+0x6D) [0x7f18026a7b2d] ----- END BACKTRACE -----
kaloian.manassiev commented on Mon, 29 Jun 2015 13:05:07 +0000: david.golden, let me know if it went away with the latest build. xgen-internal-githook commented on Sat, 27 Jun 2015 18:02:06 +0000: Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-19004 Get rid of Shard::runCommand Branch: master https://github.com/mongodb/mongo/commit/38f937036b5033bd50a9fd740e897415bd9f21db behackett commented on Sat, 27 Jun 2015 16:41:38 +0000: Reopening since the commit that resolved this ticket has been reverted. xgen-internal-githook commented on Sat, 27 Jun 2015 01:06:11 +0000: Author: {u'username': u'visualzhou', u'name': u'Siyuan Zhou', u'email': u'visualzhou@gmail.com'} Message: Revert "SERVER-19004 Get rid of Shard::runCommand" This reverts commit ce22d7ea21ad0f529b0dbb4e0b9264d1a68637ff. Branch: master https://github.com/mongodb/mongo/commit/8910277205b7b1cc52e4650c38172f801c522597 kaloian.manassiev commented on Fri, 26 Jun 2015 22:18:43 +0000: Should be fixed by this commit: https://github.com/mongodb/mongo/commit/ce22d7ea21ad0f529b0dbb4e0b9264d1a68637ff xgen-internal-githook commented on Fri, 26 Jun 2015 22:17:15 +0000: Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-19004 Get rid of Shard::runCommand This change removes the ability to call runCommand directly through the Shard class and instead will require all callers to go through the targeter and the shard registry. It also special cases running commands on the config shard to go through the catalog manager. Branch: master https://github.com/mongodb/mongo/commit/ce22d7ea21ad0f529b0dbb4e0b9264d1a68637ff behackett commented on Thu, 18 Jun 2015 05:56:33 +0000: The problem persists with the latest nightly, which includes the above commit. I'm attaching a new logfile. The backtrace looks the same: 2015-06-18T05:49:31.484+0000 D - [main] tcmallocPoolSize: 1073741824 2015-06-18T05:49:31.495+0000 I CONTROL [main] 2015-06-18T05:49:31.495+0000 I CONTROL [main] ** NOTE: This is a development version (3.1.5-pre-) of MongoDB. 2015-06-18T05:49:31.495+0000 I CONTROL [main] ** Not recommended for production. 2015-06-18T05:49:31.495+0000 I CONTROL [main] 2015-06-18T05:49:31.495+0000 I SHARDING [mongosMain] MongoS version 3.1.5-pre- starting: pid=1986 port=27017 64-bit host=ip-10-167-153-159 (--help for usage) 2015-06-18T05:49:31.495+0000 I CONTROL [mongosMain] db version v3.1.5-pre- 2015-06-18T05:49:31.495+0000 I CONTROL [mongosMain] git version: e383a423d98daade18179738d72862cf6946580b 2015-06-18T05:49:31.495+0000 I CONTROL [mongosMain] OpenSSL version: OpenSSL 1.0.0-fips 29 Mar 2010 2015-06-18T05:49:31.495+0000 I CONTROL [mongosMain] allocator: tcmalloc 2015-06-18T05:49:48.740+0000 D SHARDING [conn17] created new distributed lock for pymongo_test_mike on localhost:27016,localhost:27021,localhost:27022 ( lock timeout : 900000, ping interval : 30000, process : 0 ) 2015-06-18T05:49:48.740+0000 D SHARDING [conn17] trying to acquire new distributed lock for pymongo_test_mike on localhost:27016,localhost:27021,localhost:27022 ( lock timeout : 900000, ping interval : 30000, process : ip-10-167-153-159:27017:1434606571:1804289383 ) 2015-06-18T05:49:48.740+0000 D SHARDING [conn17] inserting initial doc in config.locks for lock pymongo_test_mike 2015-06-18T05:49:49.042+0000 D SHARDING [conn17] about to acquire distributed lock 'pymongo_test_mike/ip-10-167-153-159:27017:1434606571:1804289383' 2015-06-18T05:49:49.042+0000 D SHARDING [conn17] trying to acquire lock { _id: "pymongo_test_mike", state: 0 } with details { state: 1, who: "ip-10-167-153-159:27017:1434606571:1804289383:conn17:1957747793", process: "ip-10-167-153-159:27017:1434606571:1804289383", when: new Date(1434606589042), why: "createDatabase", ts: ObjectId('55825bfd331435ae12ad0598') } 2015-06-18T05:49:49.647+0000 I SHARDING [conn17] distributed lock 'pymongo_test_mike/ip-10-167-153-159:27017:1434606571:1804289383' acquired, ts : 55825bfd331435ae12ad0598 2015-06-18T05:49:49.647+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27020, no events 2015-06-18T05:49:49.648+0000 D SHARDING [conn17] best shard for new allocation is dataSizeBytes: 838860800 version: 3.1.5-pre- 2015-06-18T05:49:49.648+0000 I SHARDING [conn17] Placing [pymongo_test_mike] on: sh01:localhost:27020 2015-06-18T05:49:49.648+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27016, no events 2015-06-18T05:49:49.648+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27021, no events 2015-06-18T05:49:49.648+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27022, no events 2015-06-18T05:49:49.699+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27016, no events 2015-06-18T05:49:49.699+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27021, no events 2015-06-18T05:49:49.699+0000 D NETWORK [conn17] polling for status of connection to 127.0.0.1:27022, no events 2015-06-18T05:49:50.002+0000 I SHARDING [conn17] distributed lock 'pymongo_test_mike/ip-10-167-153-159:27017:1434606571:1804289383' unlocked. 2015-06-18T05:49:50.002+0000 D SHARDING [conn17] found 0 collections left and 0 collections dropped for database pymongo_test_mike 2015-06-18T05:49:50.349+0000 F - [conn17] Invalid access at address: 0 2015-06-18T05:49:50.374+0000 F - [conn17] Got signal: 11 (Segmentation fault). 0xa9c316 0xa9ba32 0xa9bd6e 0x7f2b024cc130 0x9734c6 0x974a9b 0x9a04d7 0x9e2668 0x9e322c 0x9e8e38 0x9e1c30 0x5fff9e 0xa55021 0x7f2b024c4df3 0x7f2b021f21ad ----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"400000","o":"69C316"},{"b":"400000","o":"69BA32"},{"b":"400000","o":"69BD6E"},{"b":"7F2B024BD000","o":"F130"},{"b":"400000","o":"5734C6"},{"b":"400000","o":"574A9B"},{"b":"400000","o":"5A04D7"},{"b":"400000","o":"5E2668"},{"b":"400000","o":"5E322C"},{"b":"400000","o":"5E8E38"},{"b":"400000","o":"5E1C30"},{"b":"400000","o":"1FFF9E"},{"b":"400000","o":"655021"},{"b":"7F2B024BD000","o":"7DF3"},{"b":"7F2B020FC000","o":"F61AD"}],"processInfo":{ "mongodbVersion" : "3.1.5-pre-", "gitVersion" : "e383a423d98daade18179738d72862cf6946580b", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.14.44-32.39.amzn1.x86_64", "version" : "#1 SMP Thu Jun 11 20:33:38 UTC 2015", "machine" : "x86_64" }, "somap" : [ { "elfType" : 2, "b" : "400000", "buildId" : "0C6F5FC153EB0C894E1D96D0806653EFE79B7505" }, { "b" : "7FFC4A0EB000", "elfType" : 3, "buildId" : "3C9325999495818299EC622C63D9330D234E02C9" }, { "b" : "7F2B03B96000", "path" : "/usr/lib64/libsasl2.so.2", "elfType" : 3, "buildId" : "354560FFC93703E5A80EEC8C66DF9E59DA335001" }, { "b" : "7F2B03953000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "DE5A9F7A11A0881CB64E375F4DDCA58028F0FAF8" }, { "b" : "7F2B036E6000", "path" : "/usr/lib64/libssl.so.10", "elfType" : 3, "buildId" : "22480480235F3B1C6C2E5E5953949728676D3796" }, { "b" : "7F2B03301000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "ADD80D7DBE8B04C3BA8E3242D96F39FF870A862A" }, { "b" : "7F2B030F9000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "E81013CBFA409053D58A65A0653271AB665A4619" }, { "b" : "7F2B02EF5000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "62A8842157C62F95C3069CBF779AFCC26577A99A" }, { "b" : "7F2B02BF1000", "path" : "/usr/lib64/libstdc++.so.6", "elfType" : 3, "buildId" : "DD6383EEAC49E9BAA9E3D1080AE932F42CF8A385" }, { "b" : "7F2B028EF000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "5F97F8F8E5024E29717CF35998681F84D4A22D45" }, { "b" : "7F2B026D9000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "C52958E393BDF8E8D090F36DE0F4E620D8736FBF" }, { "b" : "7F2B024BD000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "D48D3E6672A77B603B402F661BABF75E90AD570B" }, { "b" : "7F2B020FC000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "A14FC690F08FB799BA8CC82D49DE9AA9D4580464" }, { "b" : "7F2B03DB1000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "6F90843B9087FE91955FEB0355EB0858EF9E97B2" }, { "b" : "7F2B01EE2000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "6A7DA1CED90F65F27CB7B5BACDBB1C386C05F592" }, { "b" : "7F2B01CAB000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "0FBEA014763DF743B9C1DCB73EBA5E92B8B6FA35" }, { "b" : "7F2B019C6000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "A3E43FC66908AC8B00773707FECA3B1677AFF311" }, { "b" : "7F2B0179B000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "B10FBFEC246C4EAD1719D16090D0BE54904BBFC9" }, { "b" : "7F2B01598000", "path" : "/usr/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "622F315EB5CB2F791E9B64020692EBA98195D06D" }, { "b" : "7F2B0138D000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "7292C0673D7C116E3389D3FFA67087A6B9287A71" }, { "b" : "7F2B0118A000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "37A58210FA50C91E09387765408A92909468D25B" }, { "b" : "7F2B00F74000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "89C6AF118B6B4FB6A73AE1813E2C8BDD722956D1" }, { "b" : "7F2B00CF9000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "F32A30168D31318D17B95A69D9EE3D7181027A6C" }, { "b" : "7F2B00AD8000", "path" : "/usr/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "F5054DC94443326819FBF3065CFDF5E4726F57EE" }, { "b" : "7F2AFA3AB000", "path" : "/usr/lib64/sasl2/libsasldb.so", "elfType" : 3, "buildId" : "6B82ECC04F262F87646E106FD82254D7B25684A6" }, { "b" : "7F2AFA03C000", "path" : "/lib64/libdb-4.7.so", "elfType" : 3, "buildId" : "E3E19783ECE59CE43F5C2EAB3F2529DFE798EB3B" }, { "b" : "7F2AF9E37000", "path" : "/usr/lib64/sasl2/libplain.so", "elfType" : 3, "buildId" : "916D3F3990557E30CB579ED3D144FA57774B9B6C" }, { "b" : "7F2AF9C32000", "path" : "/usr/lib64/sasl2/liblogin.so", "elfType" : 3, "buildId" : "7660C66EFF41AB59B1740F97C3E6F8D79D23567C" }, { "b" : "7F2AF9A2D000", "path" : "/usr/lib64/sasl2/libanonymous.so", "elfType" : 3, "buildId" : "523C058C8D946374FF6651FE748617BAC2A37371" } ] }} mongos(_ZN5mongo15printStackTraceERSo+0x26) [0xa9c316] mongos(+0x69BA32) [0xa9ba32] mongos(+0x69BD6E) [0xa9bd6e] libpthread.so.0(+0xF130) [0x7f2b024cc130] mongos(_ZNK5mongo5Shard10runCommandERKSsRKNS_7BSONObjERS3_+0x66) [0x9734c6] mongos(_ZNK5mongo5Shard10runCommandERKSsS2_RNS_7BSONObjE+0x20B) [0x974a9b] mongos(+0x5A04D7) [0x9a04d7] mongos(_ZN5mongo7Command22execCommandClientBasicEPNS_16OperationContextEPS0_RNS_11ClientBasicEiPKcRNS_7BSONObjERNS_14BSONObjBuilderE+0x4B8) [0x9e2668] mongos(_ZN5mongo7Command20runAgainstRegisteredEPKcRNS_7BSONObjERNS_14BSONObjBuilderEi+0x26C) [0x9e322c] mongos(_ZN5mongo8Strategy15clientCommandOpERNS_7RequestE+0x1B8) [0x9e8e38] mongos(_ZN5mongo7Request7processEi+0x690) [0x9e1c30] mongos(_ZN5mongo21ShardedMessageHandler7processERNS_7MessageEPNS_21AbstractMessagingPortE+0x3E) [0x5fff9e] mongos(_ZN5mongo17PortMessageServer17handleIncomingMsgEPv+0x261) [0xa55021] libpthread.so.0(+0x7DF3) [0x7f2b024c4df3] libc.so.6(clone+0x6D) [0x7f2b021f21ad] ----- END BACKTRACE ----- behackett commented on Wed, 17 Jun 2015 18:24:03 +0000: That was fast! I'll post back if the next nightly continues to have issues. xgen-internal-githook commented on Wed, 17 Jun 2015 18:20:57 +0000: Author: {u'username': u'kaloianm', u'name': u'Kaloian Manassiev', u'email': u'kaloian.manassiev@mongodb.com'} Message: SERVER-18464/SERVER-18437/SERVER-19004 Move targeter ownership to Shard Currently there is a discrepancy between the ownership of the targeter object and the lifetime of Shard. It is possible that if a shard has been created and a targeter disappears then operations on that shard will not find the targeter. This change moves the ownership of targeter to the Shard object itself. Branch: master https://github.com/mongodb/mongo/commit/6584bfd3ddcdb1598fb486615c49f0272798f5d6 behackett commented on Wed, 17 Jun 2015 01:59:51 +0000: The command sent to mongo-orchestration to start the cluster: curl -i -H Accept: application/json -X POST -d { "routers": [{"setParameter" : {"enableTestCommands" : 1}, "vv" : true, "port": 27017, "logpath": "/mnt/jenkins/log/router27017.log"}, {"setParameter" : {"enableTestCommands" : 1}, "vv" : true, "port": 27018, "logpath": "/mnt/jenkins/log/router27018.log"}], "configsvrs": [{"port": 27016, "dbpath": "/mnt/jenkins/data/db27016", "logpath": "/mnt/jenkins/log/configsvr27016.log"}, {"port": 27021, "dbpath": "/mnt/jenkins/data/db27021", "logpath": "/mnt/jenkins/log/configsvr27021.log"}, {"port": 27022, "dbpath": "/mnt/jenkins/data/db27022", "logpath": "/mnt/jenkins/log/configsvr27022.log"}], "id": "shard_cluster_1", "shards": [{"id": "sh01", "shardParams": {"procParams": {"setParameter" : {"enableTestCommands" : 1}, "vv" : true, "storageEngine": "mmapv1", "port": 27020, "dbpath": "/mnt/jenkins/data/db27020", "logpath":"/mnt/jenkins/log/db27020.log", "ipv6":true, "logappend":true, "nojournal": true}}}]} http://localhost:8889/sharded_clusters