Info

A 3.6.3 enterprise version mongod instance using LDAP has crashed several times, at fairly regular intervals (4 and bit hours apart). The backtrace in each case is always like the below. Unable to find source-code formatter for language: txt. Available languages are: actionscript, ada, applescript, bash, c, c#, c++, cpp, css, erlang, go, groovy, haskell, html, java, javascript, js, json, lua, none, nyan, objc, perl, php, python, r, rainbow, ruby, scala, sh, sql, swift, visualbasic, xml, yaml 2018-05-06T01:05:02.282-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin 2018-05-06T01:05:22.289-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin 2018-05-06T01:05:32.289-0400 I ACCESS [conn5] Successfully authenticated as principal mms-monitoring-agent on admin 2018-05-06T01:05:40.463-0400 I ACCESS [conn368] Successfully authenticated as principal mms-monitoring-agent on admin 2018-05-06T01:05:46.918-0400 F - [conn3] Failed to mlock: Cannot allocate memory 2018-05-06T01:05:46.918-0400 F - [conn3] Fatal Assertion 28832 at src/mongo/base/secure_allocator.cpp 246 2018-05-06T01:05:46.918-0400 F - [conn3] ***aborting after fassert() failure 2018-05-06T01:05:46.934-0400 F - [conn3] Got signal: 6 (Aborted). 0x5fd75550 0x5fd74374 0x5fd749d4 0x3fffb1cf0478 0x3fffb0cf1f94 0x5deb322c 0x5fcd435c 0x5f7af07c 0x5f7afe34 0x5f7a65f4 0x5f799e14 0x5f79aba8 0x5e479cc4 0x5f7fd648 0x5f8005d0 0x5f813494 0x5e12157c 0x5e130184 0x5e12a954 0x5e12e4ac 0x5f527958 0x5e129134 0x5e12be24 0x5e12c8f4 0x5e12a9f8 0x5e12e4ac 0x5f528048 0x5fba7848 0x3fffb0ea8af4 0x3fffb0dd8814 mongod.log.2018-05-06T05-05-56:----- BEGIN BACKTRACE ----- {"backtrace":[{"b":"371E0000","o":"2865550","s":"_ZN5mongo15printStackTraceERSo"},{"b":"371E0000","o":"2864374"},{"b":"371E0000","o":"28649D4"},{"b":"3FFF9E650000","o":"478","s":"__kernel_sigtramp_rt64"},{"b":"3FFF9D610000","o":"41F94","s":"abort"},{"b":"371E0000","o":"9A322C","s":"_ZN5mongo25fassertFailedWithLocationEiPKcj"},{"b":"371E0000","o":"27C435C","s":"_ZN5mongo24secure_allocator_details8allocateEmm"},{"b":"371E0000","o":"229F07C"},{"b":"371E0000","o":"229FE34","s":"_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE"},{"b":"371E0000","o":"22965F4","s":"_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE"},{"b":"371E0000","o":"2289E14","s":"_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE"},{"b":"371E0000","o":"228ABA8","s":"_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE"},{"b":"371E0000","o":"F69CC4","s":"_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE"},{"b":"371E0000","o":"22ED648","s":"_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE"},{"b":"371E0000","o":"22F05D0","s":"_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE"},{"b":"371E0000","o":"2303494","s":"_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE"},{"b":"371E0000","o":"C1157C","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE"},{"b":"371E0000","o":"C20184","s":"_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A954","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2017958","s":"_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE"},{"b":"371E0000","o":"C19134","s":"_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE"},{"b":"371E0000","o":"C1BE24","s":"_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE"},{"b":"371E0000","o":"C1C8F4","s":"_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1A9F8","s":"_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE"},{"b":"371E0000","o":"C1E4AC"},{"b":"371E0000","o":"2018048"},{"b":"371E0000","o":"2697848"},{"b":"3FFF9D800000","o":"8AF4"},{"b":"3FFF9D610000","o":"128814","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.6.3", "gitVersion" : "9586e557d54ef70f9ca4b43c26892cd55257e1a5", "compiledModules" : [ "enterprise" ], "uname" : { "sysname" : "Linux", "release" : "3.10.0-693.17.1.el7.ppc64le", "version" : "#1 SMP Sun Jan 14 10:42:02 EST 2018", "machine" : "ppc64le" }, "somap" : [ { "b" : "371E0000", "elfType" : 3, "buildId" : "CB33058B673BEF7BA160427807DC1C100B94D6BA" }, { "b" : "3FFF9E650000", "elfType" : 3, "buildId" : "2C75A99E1CFE5BC193FFFC0EBAD200C84A78B68D" }, { "b" : "3FFF9E5A0000", "path" : "/lib64/libcurl.so.4", "elfType" : 3, "buildId" : "D527B09409D52E9BFAE267B39C0FDF3D38E759A3" }, { "b" : "3FFF9E280000", "path" : "/lib64/libnetsnmpmibs.so.31", "elfType" : 3, "buildId" : "077A8F38393AE25F667612B8B59E9E77E409B52E" }, { "b" : "3FFF9E240000", "path" : "/lib64/libsensors.so.4", "elfType" : 3, "buildId" : "54CC83D389C6B65799697562CA1DD3B6B02EE4E0" }, { "b" : "3FFF9E210000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "969ECD5474260DBCB4B2ABFB31F51C064AC4C6E9" }, { "b" : "3FFF9E170000", "path" : "/lib64/librpm.so.3", "elfType" : 3, "buildId" : "CB8D493850D9B9E3BAFC7DF944F9C0945AD24B19" }, { "b" : "3FFF9E110000", "path" : "/lib64/librpmio.so.3", "elfType" : 3, "buildId" : "02DC64D69171CFCD3BE7346187970A1FA8F0FAC6" }, { "b" : "3FFF9E060000", "path" : "/lib64/libnetsnmpagent.so.31", "elfType" : 3, "buildId" : "089A3522498B9D14B0C252F4F50BE5A074799CB0" }, { "b" : "3FFF9E030000", "path" : "/lib64/libwrap.so.0", "elfType" : 3, "buildId" : "91974904A33B9779D30DA471FFE9057E7628C174" }, { "b" : "3FFF9DEE0000", "path" : "/lib64/libnetsnmp.so.31", "elfType" : 3, "buildId" : "FDFA734AB43A5C39A89FFB8F6D0C72C53824A8FC" }, { "b" : "3FFF9DE40000", "path" : "/lib64/libssl.so.10", "elfType" : 3, "buildId" : "6E0617AAB5BA600C362C0CAE397427AE26C729EB" }, { "b" : "3FFF9DB80000", "path" : "/lib64/libcrypto.so.10", "elfType" : 3, "buildId" : "2A87A1B2F916FB15784402846384B46628C6AD79" }, { "b" : "3FFF9DB30000", "path" : "/lib64/libsasl2.so.3", "elfType" : 3, "buildId" : "3F93014C00BC39CE2CD5C38BFA246ED8E69ECD37" }, { "b" : "3FFF9DAA0000", "path" : "/lib64/libldap-2.4.so.2", "elfType" : 3, "buildId" : "0988B930AA8F850B430045189134AD745A266E0C" }, { "b" : "3FFF9DA60000", "path" : "/lib64/liblber-2.4.so.2", "elfType" : 3, "buildId" : "E82B0444E3C3C2A4F61D890764E5BE0DC533122F" }, { "b" : "3FFF9D9E0000", "path" : "/lib64/libgssapi_krb5.so.2", "elfType" : 3, "buildId" : "2C96E0BE57939D6BAF13E0BFB1B5BF8EBA59EC0C" }, { "b" : "3FFF9D8F0000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "68BDE3292C76B915D2D754B412C1EBC04F577651" }, { "b" : "3FFF9D8B0000", "path" : "/lib64/libresolv.so.2", "elfType" : 3, "buildId" : "7812741EA4E2159DC508E42898665472C6975EB0" }, { "b" : "3FFF9D880000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "03B6F7D0C119E4E73F4FECA9348935B6FDD0C81C" }, { "b" : "3FFF9D840000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "CDC8EA09CEE9BD7C6FFCA5B32B6751034349039E" }, { "b" : "3FFF9D800000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "CB2DE03746A9E9FF42316A9B913624EFB7CC206E" }, { "b" : "3FFF9D610000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "9DAEB87441B1A97240F1F46F69886764BCADC31D" }, { "b" : "3FFF9E670000", "path" : "/lib64/ld64.so.2", "elfType" : 3, "buildId" : "6339F0F333D1B8C587C27E4E18F4A13C68E3A75E" }, { "b" : "3FFF9D5B0000", "path" : "/lib64/libidn.so.11", "elfType" : 3, "buildId" : "1FBC9A52A46AFC63447524B7FC20DD6B6317DB9A" }, { "b" : "3FFF9D550000", "path" : "/lib64/libssh2.so.1", "elfType" : 3, "buildId" : "68B0A2A707ECE31BA3A5A8681B62BE3C4F262580" }, { "b" : "3FFF9D4D0000", "path" : "/lib64/libssl3.so", "elfType" : 3, "buildId" : "02DC767EC098B356F39CD6B5040DA5D821187934" }, { "b" : "3FFF9D480000", "path" : "/lib64/libsmime3.so", "elfType" : 3, "buildId" : "1CB51DF2FC8DDBA92BD1550B7EC377B38B52C013" }, { "b" : "3FFF9D300000", "path" : "/lib64/libnss3.so", "elfType" : 3, "buildId" : "89B4886B985A919B1772A06E146390D44D1149A0" }, { "b" : "3FFF9D2A0000", "path" : "/lib64/libnssutil3.so", "elfType" : 3, "buildId" : "736ECBC237DE290F1EA3F96814DE4B252DFDFD35" }, { "b" : "3FFF9D270000", "path" : "/lib64/libplds4.so", "elfType" : 3, "buildId" : "EF7B4CAD945B5ACC48EBA382DC5BDAF648CB260A" }, { "b" : "3FFF9D240000", "path" : "/lib64/libplc4.so", "elfType" : 3, "buildId" : "831682C943C06075EE5E785631A739D4811C207E" }, { "b" : "3FFF9D1D0000", "path" : "/lib64/libnspr4.so", "elfType" : 3, "buildId" : "1EF3D9EAAC25BD1AE78BFAE40BE1BF2FC9EF1A86" }, { "b" : "3FFF9D0A0000", "path" : "/lib64/libkrb5.so.3", "elfType" : 3, "buildId" : "B55BD49FCC7D957739A3236377538F706E757D4A" }, { "b" : "3FFF9D040000", "path" : "/lib64/libk5crypto.so.3", "elfType" : 3, "buildId" : "C1BAEA99E6FD27FA77BCB77C26B45184DC86F2A6" }, { "b" : "3FFF9D010000", "path" : "/lib64/libcom_err.so.2", "elfType" : 3, "buildId" : "D4B216A44EB7603EFA730565106921DB8F92A124" }, { "b" : "3FFF9CFD0000", "path" : "/lib64/libz.so.1", "elfType" : 3, "buildId" : "11FCD88E67465947D0E20EEC65C8A499087EB34E" }, { "b" : "3FFF9CDF0000", "path" : "/usr/lib64/perl5/CORE/libperl.so", "elfType" : 3, "buildId" : "70D11256C4702E4FBAE6B58E876818C6A76476D5" }, { "b" : "3FFF9CDB0000", "path" : "/lib64/libnsl.so.1", "elfType" : 3, "buildId" : "9CCD982ECDFB582090DA0FFB4C40443F2E331ECF" }, { "b" : "3FFF9CD60000", "path" : "/lib64/libcrypt.so.1", "elfType" : 3, "buildId" : "3ED0390673BCCA3EA49854F13CBBFDDC2EC83409" }, { "b" : "3FFF9CD30000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "8CDCAC8C2941127351284ADB8CC0C9D2A005CF8D" }, { "b" : "3FFF9CCF0000", "path" : "/lib64/libbz2.so.1", "elfType" : 3, "buildId" : "ADF50408432617067AD8B3D062CD57608A3CB2A6" }, { "b" : "3FFF9CCB0000", "path" : "/lib64/libelf.so.1", "elfType" : 3, "buildId" : "DDB70F20DE3E4632C3343AF17394D4ED2339622E" }, { "b" : "3FFF9CC50000", "path" : "/lib64/liblzma.so.5", "elfType" : 3, "buildId" : "1DAA639AA47F757D3637C04E54DDCE183823BD2D" }, { "b" : "3FFF9CC20000", "path" : "/lib64/libpopt.so.0", "elfType" : 3, "buildId" : "AFD04B59E402943412B1027AE2219299FEA875C8" }, { "b" : "3FFF9CBD0000", "path" : "/lib64/libselinux.so.1", "elfType" : 3, "buildId" : "71FA3939566953509C7FE36A83C73B9FE22A85CC" }, { "b" : "3FFF9CBA0000", "path" : "/lib64/libcap.so.2", "elfType" : 3, "buildId" : "70879B51ACAF692786A561A359D86D8D89EB861E" }, { "b" : "3FFF9CB60000", "path" : "/lib64/libacl.so.1", "elfType" : 3, "buildId" : "A1C181B034E65CFE4B12095775CAD9274B9BDF1B" }, { "b" : "3FFF9CB00000", "path" : "/lib64/liblua-5.1.so", "elfType" : 3, "buildId" : "871E8C147E5F3FA59D568CE41778920E3C771FDE" }, { "b" : "3FFF9C8F0000", "path" : "/lib64/libdb-5.3.so", "elfType" : 3, "buildId" : "A2B813A140BBBD02EB8935651719A22B5D82E736" }, { "b" : "3FFF9C8C0000", "path" : "/lib64/libkrb5support.so.0", "elfType" : 3, "buildId" : "CCDA94380A7D7CC3EE0D311E2EA34267E9EEE809" }, { "b" : "3FFF9C890000", "path" : "/lib64/libkeyutils.so.1", "elfType" : 3, "buildId" : "E6FCF4AFACA8AE007CDB4419DF42040D1C0E8CA0" }, { "b" : "3FFF9C860000", "path" : "/lib64/libfreebl3.so", "elfType" : 3, "buildId" : "43D98F1422C06A50678FEE88DB20A1067D77A449" }, { "b" : "3FFF9C7F0000", "path" : "/lib64/libpcre.so.1", "elfType" : 3, "buildId" : "88B77E47D365CB47FE391B128FD52B84868AD027" }, { "b" : "3FFF9C7C0000", "path" : "/lib64/libattr.so.1", "elfType" : 3, "buildId" : "8510E155A4490CFAE6014E03D89C0231541CD3E3" } ] }} mongod(_ZN5mongo15printStackTraceERSo+0x80) [0x39a45550] mongod(+0x2864374) [0x39a44374] mongod(+0x28649D4) [0x39a449d4] (__kernel_sigtramp_rt64+0x0) [0x3fff9e650478] libc.so.6(abort+0x2B4) [0x3fff9d651f94] mongod(_ZN5mongo25fassertFailedWithLocationEiPKcj+0x1C0) [0x37b8322c] mongod(_ZN5mongo24secure_allocator_details8allocateEmm+0x77C) [0x399a435c] mongod(+0x229F07C) [0x3947f07c] mongod(_ZN5mongo18OpenLDAPConnection10bindAsUserERKNS_15LDAPBindOptionsE+0x64) [0x3947fe34] mongod(_ZN5mongo14LDAPRunnerImpl8runQueryB5cxx11ERKNS_9LDAPQueryE+0x224) [0x394765f4] mongod(_ZN5mongo15LDAPManagerImpl22_getGroupDNsFromServerB5cxx11ERNS_9LDAPQueryE+0x74) [0x39469e14] mongod(_ZN5mongo15LDAPManagerImpl12getUserRolesERKNS_8UserNameE+0x528) [0x3946aba8] mongod(_ZN5mongo29AuthzManagerExternalStateLDAP18getUserDescriptionEPNS_16OperationContextERKNS_8UserNameEPNS_7BSONObjE+0x134) [0x38149cc4] mongod(_ZN5mongo20AuthorizationManager12_fetchUserV2EPNS_16OperationContextERKNS_8UserNameEPSt10unique_ptrINS_4UserESt14default_deleteIS7_EE+0x88) [0x394cd648] mongod(_ZN5mongo20AuthorizationManager11acquireUserEPNS_16OperationContextERKNS_8UserNameEPPNS_4UserE+0x620) [0x394d05d0] mongod(_ZN5mongo20AuthorizationSession24_refreshUserInfoAsNeededEPNS_16OperationContextE+0x204) [0x394e3494] mongod(_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE+0xDC) [0x37df157c] mongod(_ZN5mongo19ServiceStateMachine15_processMessageENS0_11ThreadGuardE+0x104) [0x37e00184] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0xC4) [0x37dfa954] mongod(+0xC1E4AC) [0x37dfe4ac] mongod(_ZN5mongo9transport26ServiceExecutorSynchronous8scheduleESt8functionIFvvEENS0_15ServiceExecutor13ScheduleFlagsE+0x208) [0x391f7958] mongod(_ZN5mongo19ServiceStateMachine22_scheduleNextWithGuardENS0_11ThreadGuardENS_9transport15ServiceExecutor13ScheduleFlagsENS0_9OwnershipE+0x1C4) [0x37df9134] mongod(_ZN5mongo19ServiceStateMachine15_sourceCallbackENS_6StatusE+0xB04) [0x37dfbe24] mongod(_ZN5mongo19ServiceStateMachine14_sourceMessageENS0_11ThreadGuardE+0x354) [0x37dfc8f4] mongod(_ZN5mongo19ServiceStateMachine15_runNextInGuardENS0_11ThreadGuardE+0x168) [0x37dfa9f8] mongod(+0xC1E4AC) [0x37dfe4ac] mongod(+0x2018048) [0x391f8048] mongod(+0x2697848) [0x39877848] libpthread.so.0(+0x8AF4) [0x3fff9d808af4] libc.so.6(clone+0xE4) [0x3fff9d738814] ----- END BACKTRACE ----- The 'max locked memory' limit in effect on this mongod process was 64kb. The LDAP related options it was start with { ..., security: { authorization: "enabled", keyFile: "...", ldap: { authz: { queryTemplate: "{USER}?memberOf?base" }, bind: { method: "simple", queryPassword: "", queryUser: "CN=...,OU=...,OU=...,DC=...,DC=..." }, servers: "....", transportSecurity: "none", userToDNMapping: "[ { match : "(.+)", ldapQuery: "DC=...,DC=...??sub?(saMAccountName={0})" } ]" } }, setParameter: { authenticationMechanisms: "SCRAM-SHA-1,PLAIN" }, ... } Can we understand if using LDAP means we need to raise the amount of available locked memory? Or can this be fixed to prevent >= 64kb of locked memory being used? At first this error looked like SERVER-29086, SERVER-28997, but those cases didn't use LDAP and should be fixed in 3.6.0+.

Top User Comments

akira.kurogane commented on Mon, 7 May 2018 06:26:01 +0000: An extra observation: the crashes in this environment are happening 4.2hrs after each restart during the last 48hrs. Maybe exactly to the second - the error factor seems to be within a few ten's of seconds at most. In the 48hrs before that time it was variable, between 7.0 and 8.4 hrs of uptime before each crash.

Steps to Reproduce