Loading...
Loading...
We are running 6.0.4 on Ubuntu 22.04 jammy. We got an alert earlier from our app saying it lost connection to a local mongos router. Mongos.log shows the following: {"t":\{"$date":"2023-03-09T08:21:58.772-05:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn121","msg":"Writing fatal message","attr":{"message":"Invalid access at address: 0\n"}} {"t":\{"$date":"2023-03-09T08:21:58.772-05:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn121","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).\n"}} {"t":\{"$date":"2023-03-09T08:21:58.772-05:00"} ,"s":"F", "c":"ASSERT", "id":23079, "ctx":"conn127","msg":"Invariant failure","attr":{"expr":"!callback","file":"src/mongo/util/future_impl.h","line":443}} {"t":\{"$date":"2023-03-09T08:21:58.772-05:00"} ,"s":"F", "c":"ASSERT", "id":23080, "ctx":"conn127","msg":"\n\n***aborting after invariant() failure\n\n"} This is the first time I've ever seen this error. I have saved the mongos.log and diagnostic data and can upload it for analysis.
JIRAUSER1271630 commented on Mon, 13 Mar 2023 13:39:49 +0000: Hi Jason, I did not see any core dump file in /var/crash. I’m not sure why not though. Are you aware of any setting I should enable to make core dumps get produced? jason.chan commented on Mon, 13 Mar 2023 13:34:15 +0000: hi amit.gupta@opensense.com, thanks for the logs and diagnostic data. Would you have core dumps available or be able to capture them for any future crashes? If so, chris.kelly@mongodb.com, would you be able to create another secure portal for Amit to upload that data? JIRAUSER1271630 commented on Fri, 10 Mar 2023 20:01:40 +0000: Another crash of mongos router today: {"t":{"$date":"2023-03-10T14:56:38.042-05:00"},"s":"I", "c":"CONTROL", "id":31380, "ctx":"conn19293","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"5558A0239214","b":"55589CB2A000","o":"370F214","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"},{"a":"5558A023B759","b":"55589CB2A000","o":"3711759","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"5558A023558C","b":"55589CB2A000","o":"370B58C","s":"abruptQuitWithAddrSignal","s+":"EC"},{"a":"7FABECB10520","b":"7FABECACE000","o":"42520","s":"__sigaction","s+":"50"},{"a":"5558A008BB10","b":"55589CB2A000","o":"3561B10","s":"_ZNK5mongo16OperationContext18hasDeadlineExpiredEv","C":"mongo::OperationContext::hasDeadlineExpired() const","s+":"B0"},{"a":"5558A008D3FF","b":"55589CB2A000","o":"35633FF","s":"_ZN5mongo16OperationContext25checkForInterruptNoAssertEv","C":"mongo::OperationContext::checkForInterruptNoAssert()","s+":"7F"},{"a":"55589E449440","b":"55589CB2A000","o":"191F440","s":"_ZZN5mongo13Interruptible32waitForConditionOrInterruptUntilISt11unique_lockISt5mutexEZNS_14future_details15SharedStateBase4waitEPS0_EUlvE_EEbRNS_4stdx18condition_variableERT_NS_6Date_tET0_ENKUlNS0_9WakeSpeedEE0_clESG_","s+":"60"},{"a":"55589E44962D","b":"55589CB2A000","o":"191F62D","s":"_ZZN5mongo13Interruptible32waitForConditionOrInterruptUntilISt11unique_lockISt5mutexEZNS_14future_details15SharedStateBase4waitEPS0_EUlvE_EEbRNS_4stdx18condition_variableERT_NS_6Date_tET0_ENKUlSE_NS0_9WakeSpeedEE1_clESE_SG_","s+":"BD"},{"a":"55589E465FC5","b":"55589CB2A000","o":"193BFC5","s":"_ZN5mongo14future_details15SharedStateBase4waitEPNS_13InterruptibleE","C":"mongo::future_details::SharedStateBase::wait(mongo::Interruptible*)","s+":"175"},{"a":"55589E5BEAD8","b":"55589CB2A000","o":"1A94AD8","s":"_ZNK5mongo25MongosTopologyCoordinator18awaitHelloResponseEPNS_16OperationContextEN5boost8optionalINS_15TopologyVersionEEENS4_INS_6Date_tEEE","C":"mongo::MongosTopologyCoordinator::awaitHelloResponse(mongo::OperationContext*, boost::optional, boost::optional) const","s+":"2F8"},{"a":"55589E519850","b":"55589CB2A000","o":"19EF850","s":"_ZN5mongo12_GLOBAL__N_18CmdHello19runWithReplyBuilderEPNS_16OperationContextERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEERKNS_7BSONObjEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::(anonymous namespace)::CmdHello::runWithReplyBuilder(mongo::OperationContext*, std::__cxx11::basic_string, std::allocator > const&, mongo::BSONObj const&, mongo::rpc::ReplyBuilderInterface*)","s+":"200"},{"a":"55589F9ADA7E","b":"55589CB2A000","o":"2E83A7E","s":"_ZN5mongo37BasicCommandWithReplyBuilderInterface10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::BasicCommandWithReplyBuilderInterface::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s+":"BE"},{"a":"55589F9A54C1","b":"55589CB2A000","o":"2E7B4C1","s":"_ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s+":"61"},{"a":"55589F9A9539","b":"55589CB2A000","o":"2E7F539","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","C":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr, std::shared_ptr, mongo::transport::ServiceExecutor::ThreadingModel)","s+":"1C9"},{"a":"55589E57BDC7","b":"55589CB2A000","o":"1A51DC7","s":"_ZN5mongo12_GLOBAL__N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","C":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr, std::shared_ptr)","s+":"97"},{"a":"55589E580335","b":"55589CB2A000","o":"1A56335","s":"_ZN5mongo12_GLOBAL__N_117ExecCommandClient3runEv","C":"mongo::(anonymous namespace)::ExecCommandClient::run()","s+":"605"},{"a":"55589E580B6E","b":"55589CB2A000","o":"1A56B6E","s":"_ZN5mongo12_GLOBAL__N_118ParseAndRunCommand11RunAndRetry3runEv","C":"mongo::(anonymous namespace)::ParseAndRunCommand::RunAndRetry::run()","s+":"3EE"},{"a":"55589E584E3C","b":"55589CB2A000","o":"1A5AE3C","s":"_ZN5mongo13ClientCommand8_executeEv","C":"mongo::ClientCommand::_execute()","s+":"3BC"},{"a":"55589E586371","b":"55589CB2A000","o":"1A5C371","s":"_ZN5mongo13ClientCommand3runEv","C":"mongo::ClientCommand::run()","s+":"41"},{"a":"55589E586A4C","b":"55589CB2A000","o":"1A5CA4C","s":"_ZN5mongo19makeReadyFutureWithIZNOS_11future_util10AsyncStateINS_13ClientCommandEE13thenWithStateIZNS_8Strategy13clientCommandESt10shared_ptrINS_23RequestExecutionContextEEEUlPT_E_EEDaOSA_EUlvE_Li0EEENS_6FutureINS_14future_details17UnwrappedTypeImplINSt13invoke_resultISD_JEE4typeEE4typeEEESD_","C":"mongo::Future)::{lambda(auto:1*)#1}>::type>::type> mongo::makeReadyFutureWith::thenWithState)::{lambda(auto:1*)#1}>(mongo::Strategy::clientCommand(std::shared_ptr)::{lambda(auto:1*)#1}&&) &&::{lambda()#1}, 0>(mongo::Strategy::clientCommand(std::shared_ptr)::{lambda(auto:1*)#1})","s+":"3C"},{"a":"55589E587252","b":"55589CB2A000","o":"1A5D252","s":"_ZN5mongo8Strategy13clientCommandESt10shared_ptrINS_23RequestExecutionContextEE","C":"mongo::Strategy::clientCommand(std::shared_ptr)","s+":"192"},{"a":"55589E498011","b":"55589CB2A000","o":"196E011","s":"_ZN5mongo13HandleRequest13handleRequestEv","C":"mongo::HandleRequest::handleRequest()","s+":"111"},{"a":"55589E498538","b":"55589CB2A000","o":"196E538","s":"_ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationINS_10DbResponseEZZNOS9_4thenIZNS_13HandleRequest3runEvEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EEPNSI_ISB_EEE_EENS7_ISF_EEOT0_EUlS3_E_EEDaSG_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function::makeImpl::makeContinuation::then(mongo::HandleRequest::run()::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl*, {lambda()#3}*)#1}>(mongo::future_details::FutureImpl::then(mongo::HandleRequest::run()::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl*, {lambda()#3}*)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl::makeContinuation::then(mongo::HandleRequest::run()::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl*, {lambda()#3}*)#1}>(mongo::future_details::FutureImpl::then(mongo::HandleRequest::run()::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl*, {lambda()#3}*)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"88"},{"a":"55589E467BB7","b":"55589CB2A000","o":"193DBB7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"1B7"},{"a":"55589E467BB7","b":"55589CB2A000","o":"193DBB7","s":"_ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"1B7"},{"a":"55589E498E9F","b":"55589CB2A000","o":"196EE9F","s":"_ZN5mongo13HandleRequest3runEv","C":"mongo::HandleRequest::run()","s+":"3CF"},{"a":"55589E499D09","b":"55589CB2A000","o":"196FD09","s":"_ZN5mongo23ServiceEntryPointMongos17handleRequestImplEPNS_16OperationContextERKNS_7MessageE","C":"mongo::ServiceEntryPointMongos::handleRequestImpl(mongo::OperationContext*, mongo::Message const&)","s+":"1F9"},{"a":"55589E499F16","b":"55589CB2A000","o":"196FF16","s":"_ZN5mongo23ServiceEntryPointMongos13handleRequestEPNS_16OperationContextERKNS_7MessageE","C":"mongo::ServiceEntryPointMongos::handleRequest(mongo::OperationContext*, mongo::Message const&)","s+":"26"},{"a":"55589E7CCC47","b":"55589CB2A000","o":"1CA2C47","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","C":"mongo::transport::ServiceStateMachine::Impl::processMessage()","s+":"127"},{"a":"55589E7CE956","b":"55589CB2A000","o":"1CA4956","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","C":"mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)","s+":"76"},{"a":"55589E7CEDEF","b":"55589CB2A000","o":"1CA4DEF","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1_","C":"mongo::unique_function::makeImpl(mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"7F"},{"a":"55589FA9D393","b":"55589CB2A000","o":"2F73393","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl)::{lambda()#1}>(mongo::transport::ServiceExecutor::schedule(mongo::unique_function)::{lambda()#1}&&)::SpecificImpl::call()","s+":"33"},{"a":"55589FAA368B","b":"55589CB2A000","o":"2F7968B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#2}>(mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#2}&&)::SpecificImpl::call()","s+":"BB"},{"a":"55589FAA5610","b":"55589CB2A000","o":"2F7B610","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl)::{lambda()#4}>(mongo::launchServiceWorkerThread(mongo::unique_function)::{lambda()#4}&&)::SpecificImpl::call()","s+":"80"},{"a":"55589FAA56BC","b":"55589CB2A000","o":"2F7B6BC","s":"_ZN5mongo12_GLOBAL__N_17runFuncEPv","C":"mongo::(anonymous namespace)::runFunc(void*)","s+":"1C"},{"a":"7FABECB62B43","b":"7FABECACE000","o":"94B43","s":"pthread_condattr_setpshared","s+":"513"},{"a":"7FABECBF4A00","b":"7FABECACE000","o":"126A00","s":"__xmknodat","s+":"230"}],"processInfo":{"mongodbVersion":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","compiledModules":[],"uname":{"sysname":"Linux","release":"5.15.0-67-generic","version":"#74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023","machine":"x86_64"},"somap":[{"b":"55589CB2A000","elfType":3,"buildId":"2A25DCB1B8E1FE370722E05329549F78EE3622C8"},{"b":"7FABECACE000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"69389D485A9793DBE873F0EA2C93E02EFAA9AA3D"}]}}},"tags":[]} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"5558A0239214","b":"55589CB2A000","o":"370F214","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"5558A023B759","b":"55589CB2A000","o":"3711759","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"5558A023558C","b":"55589CB2A000","o":"370B58C","s":"abruptQuitWithAddrSignal","s+":"EC"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"7FABECB10520","b":"7FABECACE000","o":"42520","s":"__sigaction","s+":"50"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"5558A008BB10","b":"55589CB2A000","o":"3561B10","s":"_ZNK5mongo16OperationContext18hasDeadlineExpiredEv","C":"mongo::OperationContext::hasDeadlineExpired() const","s+":"B0"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"5558A008D3FF","b":"55589CB2A000","o":"35633FF","s":"_ZN5mongo16OperationContext25checkForInterruptNoAssertEv","C":"mongo::OperationContext::checkForInterruptNoAssert()","s+":"7F"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"55589E449440","b":"55589CB2A000","o":"191F440","s":"_ZZN5mongo13Interruptible32waitForConditionOrInterruptUntilISt11unique_lockISt5mutexEZNS_14future_details15SharedStateBase4waitEPS0_EUlvE_EEbRNS_4stdx18condition_variableERT_NS_6Date_tET0_ENKUlNS0_9WakeSpeedEE0_clESG_","s+":"60"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"55589E44962D","b":"55589CB2A000","o":"191F62D","s":"_ZZN5mongo13Interruptible32waitForConditionOrInterruptUntilISt11unique_lockISt5mutexEZNS_14future_details15SharedStateBase4waitEPS0_EUlvE_EEbRNS_4stdx18condition_variableERT_NS_6Date_tET0_ENKUlSE_NS0_9WakeSpeedEE1_clESE_SG_","s+":"BD"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"55589E465FC5","b":"55589CB2A000","o":"193BFC5","s":"_ZN5mongo14future_details15SharedStateBase4waitEPNS_13InterruptibleE","C":"mongo::future_details::SharedStateBase::wait(mongo::Interruptible*)","s+":"175"}}} {"t":{"$date":"2023-03-10T14:56:38.043-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn19293","msg":"Frame","attr":{"frame":{"a":"55589E5BEAD8","b":"55589CB2A000","o":"1A94AD8","s":"_ZNK5mongo25MongosTopologyCoordinator18awaitHelloResponseEPNS_16OperationContextEN5boost8optionalINS_15TopologyVersionEEENS4_INS_6Date_tEEE","C":"mongo::MongosTopologyCoordinator::awaitHelloResponse(mongo::OperationContext*, boost::optional, boost::optional) const","s+":"2F8"}}} JIRAUSER1271630 commented on Thu, 9 Mar 2023 20:21:24 +0000: Just had a second crash of the same server. Uploaded logs and diagnostics to mongod-replicaset_member_shard1-second-crash.tar.gz Here is the log snippet in case it's useful for SEO or someone else. {"t":{"$date":"2023-03-09T15:08:59.312-05:00"},"s":"F", "c":"CONTROL", "id":6384300, "ctx":"TimestampMonitor","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"E", "c":"CONTROL", "id":31430, "ctx":"TimestampMonitor","msg":"Error collecting stack trace","attr":{"error":"unw_get_proc_name(7F2F7051D71B): no unwind info found\nunw_get_proc_name(7F2F7051D71B): no unwind info found\n"}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31380, "ctx":"TimestampMonitor","msg":"BACKTRACE","attr":{"bt":{"backtrace":[{"a":"55CAFE8668D4","b":"55CAF9A88000","o":"4DDE8D4","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"},{"a":"55CAFE868E19","b":"55CAF9A88000","o":"4DE0E19","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55CAFE862AF6","b":"55CAF9A88000","o":"4DDAAF6","s":"abruptQuit","s+":"66"},{"a":"7F2F70537520","b":"7F2F704F5000","o":"42520","s":"__sigaction","s+":"50"},{"a":"7F2F7058BA7C","b":"7F2F704F5000","o":"96A7C","s":"pthread_kill","s+":"12C"},{"a":"7F2F70537476","b":"7F2F704F5000","o":"42476","s":"raise","s+":"16"},{"a":"7F2F7051D7F3","b":"7F2F704F5000","o":"287F3","s":"abort","s+":"D3"},{"a":"7F2F7051D71B","b":"7F2F704F5000","o":"2871B"},{"a":"7F2F7052EE96","b":"7F2F704F5000","o":"39E96","s":"__assert_fail","s+":"46"},{"a":"7F2F7058C201","b":"7F2F704F5000","o":"97201","s":"pthread_kill_other_threads_np","s+":"741"},{"a":"7F2F70588E05","b":"7F2F704F5000","o":"93E05","s":"pthread_cond_timedwait","s+":"125"},{"a":"55CAFE6E3D50","b":"55CAF9A88000","o":"4C5BD50","s":"_ZN5mongo11ClockSource21waitForConditionUntilERNS_4stdx18condition_variableENS_20BasicLockableAdapterENS_6Date_tEPNS_8WaitableE","C":"mongo::ClockSource::waitForConditionUntil(mongo::stdx::condition_variable&, mongo::BasicLockableAdapter, mongo::Date_t, mongo::Waitable*)","s+":"5F0"},{"a":"55CAFC241AB2","b":"55CAF9A88000","o":"27B9AB2","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_18PeriodicRunnerImpl15PeriodicJobImpl4_runEvEUlvE0_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","C":"std::thread::_State_impl(mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#2})::{lambda()#1}> > >::_M_run()","s+":"782"},{"a":"55CAFEA09E8F","b":"55CAF9A88000","o":"4F81E8F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F2F70589B43","b":"7F2F704F5000","o":"94B43","s":"pthread_condattr_setpshared","s+":"513"},{"a":"7F2F7061BA00","b":"7F2F704F5000","o":"126A00","s":"__xmknodat","s+":"230"}],"processInfo":{"mongodbVersion":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","compiledModules":[],"uname":{"sysname":"Linux","release":"5.15.0-67-generic","version":"#74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023","machine":"x86_64"},"somap":[{"b":"55CAF9A88000","elfType":3,"buildId":"BCB8F6C3AD4ABF41C81882BAE25C6885C6266CC4"},{"b":"7F2F704F5000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"69389D485A9793DBE873F0EA2C93E02EFAA9AA3D"}]}}},"tags":[]} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFE8668D4","b":"55CAF9A88000","o":"4DDE8D4","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFE868E19","b":"55CAF9A88000","o":"4DE0E19","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFE862AF6","b":"55CAF9A88000","o":"4DDAAF6","s":"abruptQuit","s+":"66"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F70537520","b":"7F2F704F5000","o":"42520","s":"__sigaction","s+":"50"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7058BA7C","b":"7F2F704F5000","o":"96A7C","s":"pthread_kill","s+":"12C"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F70537476","b":"7F2F704F5000","o":"42476","s":"raise","s+":"16"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7051D7F3","b":"7F2F704F5000","o":"287F3","s":"abort","s+":"D3"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7051D71B","b":"7F2F704F5000","o":"2871B"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7052EE96","b":"7F2F704F5000","o":"39E96","s":"__assert_fail","s+":"46"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7058C201","b":"7F2F704F5000","o":"97201","s":"pthread_kill_other_threads_np","s+":"741"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F70588E05","b":"7F2F704F5000","o":"93E05","s":"pthread_cond_timedwait","s+":"125"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFE6E3D50","b":"55CAF9A88000","o":"4C5BD50","s":"_ZN5mongo11ClockSource21waitForConditionUntilERNS_4stdx18condition_variableENS_20BasicLockableAdapterENS_6Date_tEPNS_8WaitableE","C":"mongo::ClockSource::waitForConditionUntil(mongo::stdx::condition_variable&, mongo::BasicLockableAdapter, mongo::Date_t, mongo::Waitable*)","s+":"5F0"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFC241AB2","b":"55CAF9A88000","o":"27B9AB2","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6threadC4IZNS3_18PeriodicRunnerImpl15PeriodicJobImpl4_runEvEUlvE0_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","C":"std::thread::_State_impl(mongo::PeriodicRunnerImpl::PeriodicJobImpl::_run()::{lambda()#2})::{lambda()#1}> > >::_M_run()","s+":"782"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"55CAFEA09E8F","b":"55CAF9A88000","o":"4F81E8F","s":"execute_native_thread_routine","s+":"F"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F70589B43","b":"7F2F704F5000","o":"94B43","s":"pthread_condattr_setpshared","s+":"513"}}} {"t":{"$date":"2023-03-09T15:08:59.355-05:00"},"s":"I", "c":"CONTROL", "id":31445, "ctx":"TimestampMonitor","msg":"Frame","attr":{"frame":{"a":"7F2F7061BA00","b":"7F2F704F5000","o":"126A00","s":"__xmknodat","s+":"230"}}} JIRAUSER1271630 commented on Thu, 9 Mar 2023 18:51:32 +0000: Thanks Chris, I've uploaded 4 .tar.gz files. mongos-crash.tar.gz - This is for the mongos router mongod-replicaset_member-crash.tar.gz mongod-replicaset_member-shard1-crash.tar.gz mongod-replicaset_member-shard2-crash.tar.gz mongod-replicaset_member-crash.tar.gz (this was actually for shard2 as well but a different server) JIRAUSER1265262 commented on Thu, 9 Mar 2023 18:21:17 +0000: Hi amit.gupta@opensense.com, I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link: the mongod logs the $dbpath/diagnostic.data directory (the contents are described here) Christopher JIRAUSER1271630 commented on Thu, 9 Mar 2023 17:00:33 +0000: Here is a stack trace from yet another crash that just happened on a different secondary in a replicaSet: {"t":\{"$date":"2023-03-09T08:51:55.263-05:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn3397","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).\n"}} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31380, "ctx":"conn3397","msg":"BACKTRACE","attr":{"bt":{"backtrace":[ {"a":"56099B1D68D4","b":"5609963F8000","o":"4DDE8D4","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"} ,{"a":"56099B1D8E19","b":"5609963F8000","o":"4DE0E19","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"56099B1D2C4C","b":"5609963F8000","o":"4DDAC4C","s":"abruptQuitWithAddrSignal","s+":"EC"},{"a":"7F9C219F9520","b":"7F9C219B7000","o":"42520","s":"sigaction","s+":"50"},{"a":"56099964AACE","b":"5609963F8000","o":"3252ACE","s":"_ZN4absl12lts_2021032418container_internal12raw_hash_setINS1_17NodeHashMapPolicyIxPN5mongo12ClientCursorEEENS0_13hash_internal4HashIxEESt8equal_toIxESaISt4pairIKxS6_EEE4findIxEENSH_8iteratorERSE_m","C":"absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::iterator absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::find(long long const&, unsigned long)","s+":"3E"},{"a":"560999647625","b":"5609963F8000","o":"324F625","s":"_ZN5mongo13CursorManager9pinCursorEPNS_16OperationContextExRKSt8functionIFvRKNS_12ClientCursorEEENS0_9AuthCheckE","C":"mongo::CursorManager::pinCursor(mongo::OperationContext*, long long, std::function const&, mongo::CursorManager::AuthCheck)","s+":"F5"},{"a":"56099911AD5D","b":"5609963F8000","o":"2D22D5D","s":"_ZN5mongo12_GLOBAL_N_110GetMoreCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::(anonymous namespace)::GetMoreCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s+":"14D"},{"a":"560999F625A1","b":"5609963F8000","o":"3B6A5A1","s":"ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s+":"61"},{"a":"560999F66619","b":"5609963F8000","o":"3B6E619","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","C":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr, std::shared_ptr, mongo::transport::ServiceExecutor::ThreadingModel)","s+":"1C9"},{"a":"56099855FB27","b":"5609963F8000","o":"2167B27","s":"_ZN5mongo12_GLOBAL_N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","C":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr, std::shared_ptr)","s+":"97"},{"a":"56099856FC26","b":"5609963F8000","o":"2177C26","s":"ZN5mongo12_GLOBAL_N_114RunCommandImpl11_runCommandEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runCommand()","s+":"146"},{"a":"5609985704B6","b":"5609963F8000","o":"21784B6","s":"_ZN5mongo12_GLOBAL_N_132RunCommandAndWaitForWriteConcern24_runCommandWithFailPointEv","C":"mongo::(anonymous namespace)::RunCommandAndWaitForWriteConcern::runCommandWithFailPoint()","s+":"86"},{"a":"56099857097C","b":"5609963F8000","o":"217897C","s":"_ZN5mongo12_GLOBAL_N_132RunCommandAndWaitForWriteConcern8_runImplEv","C":"mongo::(anonymous namespace)::RunCommandAndWaitForWriteConcern::runImpl()","s+":"3C"},{"a":"560998560BDC","b":"5609963F8000","o":"2168BDC","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl3runEv","C":"mongo::(anonymous namespace)::RunCommandImpl::run()","s+":"13C"},{"a":"560998567BB4","b":"5609963F8000","o":"216FBB4","s":"ZN5mongo12_GLOBAL_N_119ExecCommandDatabase12_commandExecEv","C":"mongo::(anonymous namespace)::ExecCommandDatabase::commandExec()","s+":"1E4"},{"a":"56099856C85F","b":"5609963F8000","o":"217485F","s":"_ZZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr):: {lambda()#2}::operator()()","s+":"43F"},{"a":"56099856E450","b":"5609963F8000","o":"2176450","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function::makeImpl::makeContinuation::then)::{lambda()#2} >(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl::makeContinuation::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"80"},{"a":"56099851EEB7","b":"5609963F8000","o":"2126EB7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"5609985723B2","b":"5609963F8000","o":"217A3B2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1","C":"auto mongo::future_details::FutureImpl::generalImpl::propagateResultTo(mongo::future_details::SharedStateImpl*) &&:: {lambda(mongo::future_details::FakeVoid&&)#1} , mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda(mongo::Status&&)#2}, mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda()#3}>(mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda(mongo::future_details::FakeVoid&&)#1}&&, mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda(mongo::Status&&)#2}&&, mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda()#3}&&)","s+":"152"},{"a":"560998562D1C","b":"5609963F8000","o":"216AD1C","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function::makeImpl::makeContinuation::then):: {lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl::makeContinuation::then)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#1}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#1}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"AC"},{"a":"56099851EEB7","b":"5609963F8000","o":"2126EB7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"56099856D1E3","b":"5609963F8000","o":"21751E3","s":"_ZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr)","s+":"1D3"},{"a":"56099856DA55","b":"5609963F8000","o":"2175A55","s":"ZN5mongo12_GLOBAL_N_116receivedCommandsESt10shared_ptrINS0_13HandleRequest16ExecutionContextEE","C":"mongo::(anonymous namespace)::receivedCommands(std::shared_ptr)","s+":"2D5"},{"a":"56099856E396","b":"5609963F8000","o":"2176396","s":"ZN5mongo12_GLOBAL_N_115CommandOpRunner3runEv","C":"mongo::(anonymous namespace)::CommandOpRunner::run()","s+":"36"},{"a":"56099856536F","b":"5609963F8000","o":"216D36F","s":"ZN5mongo23ServiceEntryPointCommon13handleRequestEPNS_16OperationContextERKNS_7MessageESt10unique_ptrIKNS0_5HooksESt14default_deleteIS8_EE","C":"mongo::ServiceEntryPointCommon::handleRequest(mongo::OperationContext*, mongo::Message const&, std::unique_ptr >)","s+":"53F"},{"a":"560998559B2F","b":"5609963F8000","o":"2161B2F","s":"_ZN5mongo23ServiceEntryPointMongod13handleRequestEPNS_16OperationContextERKNS_7MessageE","C":"mongo::ServiceEntryPointMongod::handleRequest(mongo::OperationContext*, mongo::Message const&)","s+":"4F"},{"a":"5609993BFEB7","b":"5609963F8000","o":"2FC7EB7","s":"_ZN5mongo9transport19ServiceStateMachine4Impl14processMessageEv","C":"mongo::transport::ServiceStateMachine::Impl::processMessage()","s+":"127"},{"a":"5609993C1BC6","b":"5609963F8000","o":"2FC9BC6","s":"_ZN5mongo9transport19ServiceStateMachine4Impl12startNewLoopERKNS_6StatusE","C":"mongo::transport::ServiceStateMachine::Impl::startNewLoop(mongo::Status const&)","s+":"76"},{"a":"5609993C205F","b":"5609963F8000","o":"2FCA05F","s":"_ZZN5mongo15unique_functionIFvNS_6StatusEEE8makeImplIZNS_9transport19ServiceStateMachine4Impl15scheduleNewLoopES1_EUlS1_E_EEDaOT_EN12SpecificImpl4callEOS1","C":"mongo::unique_function::makeImpl(mongo::transport::ServiceStateMachine::Impl::scheduleNewLoop(mongo::Status)::{lambda(mongo::Status)#1}&&)::SpecificImpl::call(mongo::Status&&)","s+":"7F"},{"a":"560998AACEE3","b":"5609963F8000","o":"26B4EE3","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport15ServiceExecutor8scheduleENS0_IFvNS_6StatusEEEEEUlvE_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl)::{lambda()#1} >(mongo::transport::ServiceExecutor::schedule(mongo::unique_function)::{lambda()#1}&&)::SpecificImpl::call()","s+":"33"},{"a":"56099A9D864B","b":"5609963F8000","o":"45E064B","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_9transport26ServiceExecutorSynchronous12scheduleTaskES2_NS4_15ServiceExecutor13ScheduleFlagsEEUlvE0_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl, mongo::transport::ServiceExecutor::ScheduleFlags):: {lambda()#2} >(mongo::transport::ServiceExecutorSynchronous::scheduleTask(mongo::unique_function, mongo::transport::ServiceExecutor::ScheduleFlags)::{lambda()#2}&&)::SpecificImpl::call()","s+":"BB"},{"a":"56099A9DA490","b":"5609963F8000","o":"45E2490","s":"_ZZN5mongo15unique_functionIFvvEE8makeImplIZNS_25launchServiceWorkerThreadES2_EUlvE2_EEDaOT_EN12SpecificImpl4callEv","C":"mongo::unique_function::makeImpl):: {lambda()#4} >(mongo::launchServiceWorkerThread(mongo::unique_function)::{lambda()#4}&&)::SpecificImpl::call()","s+":"80"},{"a":"56099A9DA53C","b":"5609963F8000","o":"45E253C","s":"ZN5mongo12_GLOBAL_N_17runFuncEPv","C":"mongo::(anonymous namespace)::runFunc(void*)","s+":"1C"},{"a":"7F9C21A4BB43","b":"7F9C219B7000","o":"94B43","s":"pthread_condattr_setpshared","s+":"513"},{"a":"7F9C21ADDA00","b":"7F9C219B7000","o":"126A00","s":"__xmknodat","s+":"230"}],"processInfo":{"mongodbVersion":"6.0.4","gitVersion":"44ff59461c1353638a71e710f385a566bcd2f547","compiledModules":[],"uname": {"sysname":"Linux","release":"5.15.0-67-generic","version":"#74-Ubuntu SMP Wed Feb 22 14:14:39 UTC 2023","machine":"x86_64"} ,"somap":[\{"b":"5609963F8000","elfType":3,"buildId":"BCB8F6C3AD4ABF41C81882BAE25C6885C6266CC4"},\{"b":"7F9C219B7000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"69389D485A9793DBE873F0EA2C93E02EFAA9AA3D"}]}}},"tags":[]} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"56099B1D68D4","b":"5609963F8000","o":"4DDE8D4","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"} }} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"56099B1D8E19","b":"5609963F8000","o":"4DE0E19","s":"_ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"} }} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"56099B1D2C4C","b":"5609963F8000","o":"4DDAC4C","s":"abruptQuitWithAddrSignal","s+":"EC"} }} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"7F9C219F9520","b":"7F9C219B7000","o":"42520","s":"__sigaction","s+":"50"} }} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"56099964AACE","b":"5609963F8000","o":"3252ACE","s":"_ZN4absl12lts_2021032418container_internal12raw_hash_setINS1_17NodeHashMapPolicyIxPN5mongo12ClientCursorEEENS0_13hash_internal4HashIxEESt8equal_toIxESaISt4pairIKxS6_EEE4findIxEENSH_8iteratorERSE_m","C":"absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::iterator absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::find(long long const&, unsigned long)","s+":"3E"} }} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31445, "ctx":"conn3397","msg":"Frame","attr":{"frame": {"a":"560999647625","b":"5609963F8000","o":"324F625","s":"_ZN5mongo13CursorManager9pinCursorEPNS_16OperationContextExRKSt8functionIFvRKNS_12ClientCursorEEENS0_9AuthCheckE","C":"mongo::CursorManager::pinCursor(mongo::OperationContext*, long long, std::function const&, mongo::CursorManager::AuthCheck)","s+":"F5"} }} JIRAUSER1271630 commented on Thu, 9 Mar 2023 14:56:29 +0000: We had another replicaset member die in a different shard with a similar error. Saved the logs and diagnostic data. JIRAUSER1271630 commented on Thu, 9 Mar 2023 13:57:56 +0000: We just got an alert that one of shard members died in a 5-node replicaset. Below is last lines in the mongod.log. Saved for upload as well (I had to cut off part of the error since it was too long for this comment) {"t":\{"$date":"2023-03-09T08:51:55.263-05:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn3397","msg":"Writing fatal message","attr":{"message":"Invalid access at address: 0\n"}} {"t":\{"$date":"2023-03-09T08:51:55.263-05:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"conn3397","msg":"Writing fatal message","attr":{"message":"Got signal: 11 (Segmentation fault).\n"}} {"t":\{"$date":"2023-03-09T08:51:55.433-05:00"} ,"s":"I", "c":"CONTROL", "id":31380, "ctx":"conn3397","msg":"BACKTRACE","attr":{"bt":{"backtrace":[ {"a":"56099B1D68D4","b":"5609963F8000","o":"4DDE8D4","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"} ,{"a":"56099B1D8E19","b":"5609963F8000","o":"4DE0E19","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"56099B1D2C4C","b":"5609963F8000","o":"4DDAC4C","s":"abruptQuitWithAddrSignal","s+":"EC"},{"a":"7F9C219F9520","b":"7F9C219B7000","o":"42520","s":"sigaction","s+":"50"},{"a":"56099964AACE","b":"5609963F8000","o":"3252ACE","s":"_ZN4absl12lts_2021032418container_internal12raw_hash_setINS1_17NodeHashMapPolicyIxPN5mongo12ClientCursorEEENS0_13hash_internal4HashIxEESt8equal_toIxESaISt4pairIKxS6_EEE4findIxEENSH_8iteratorERSE_m","C":"absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::iterator absl::lts_20210324::container_internal::raw_hash_set, absl::lts_20210324::hash_internal::Hash, std::equal_to, std::allocator > >::find(long long const&, unsigned long)","s+":"3E"},{"a":"560999647625","b":"5609963F8000","o":"324F625","s":"_ZN5mongo13CursorManager9pinCursorEPNS_16OperationContextExRKSt8functionIFvRKNS_12ClientCursorEEENS0_9AuthCheckE","C":"mongo::CursorManager::pinCursor(mongo::OperationContext*, long long, std::function const&, mongo::CursorManager::AuthCheck)","s+":"F5"},{"a":"56099911AD5D","b":"5609963F8000","o":"2D22D5D","s":"_ZN5mongo12_GLOBAL_N_110GetMoreCmd10Invocation3runEPNS_16OperationContextEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::(anonymous namespace)::GetMoreCmd::Invocation::run(mongo::OperationContext*, mongo::rpc::ReplyBuilderInterface*)","s+":"14D"},{"a":"560999F625A1","b":"5609963F8000","o":"3B6A5A1","s":"ZN5mongo14CommandHelpers20runCommandInvocationEPNS_16OperationContextERKNS_12OpMsgRequestEPNS_17CommandInvocationEPNS_3rpc21ReplyBuilderInterfaceE","C":"mongo::CommandHelpers::runCommandInvocation(mongo::OperationContext*, mongo::OpMsgRequest const&, mongo::CommandInvocation*, mongo::rpc::ReplyBuilderInterface*)","s+":"61"},{"a":"560999F66619","b":"5609963F8000","o":"3B6E619","s":"_ZN5mongo14CommandHelpers20runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEENS_9transport15ServiceExecutor14ThreadingModelE","C":"mongo::CommandHelpers::runCommandInvocation(std::shared_ptr, std::shared_ptr, mongo::transport::ServiceExecutor::ThreadingModel)","s+":"1C9"},{"a":"56099855FB27","b":"5609963F8000","o":"2167B27","s":"_ZN5mongo12_GLOBAL_N_120runCommandInvocationESt10shared_ptrINS_23RequestExecutionContextEES1_INS_17CommandInvocationEE","C":"mongo::(anonymous namespace)::runCommandInvocation(std::shared_ptr, std::shared_ptr)","s+":"97"},{"a":"56099856FC26","b":"5609963F8000","o":"2177C26","s":"ZN5mongo12_GLOBAL_N_114RunCommandImpl11_runCommandEv","C":"mongo::(anonymous namespace)::RunCommandImpl::runCommand()","s+":"146"},{"a":"5609985704B6","b":"5609963F8000","o":"21784B6","s":"_ZN5mongo12_GLOBAL_N_132RunCommandAndWaitForWriteConcern24_runCommandWithFailPointEv","C":"mongo::(anonymous namespace)::RunCommandAndWaitForWriteConcern::runCommandWithFailPoint()","s+":"86"},{"a":"56099857097C","b":"5609963F8000","o":"217897C","s":"_ZN5mongo12_GLOBAL_N_132RunCommandAndWaitForWriteConcern8_runImplEv","C":"mongo::(anonymous namespace)::RunCommandAndWaitForWriteConcern::runImpl()","s+":"3C"},{"a":"560998560BDC","b":"5609963F8000","o":"2168BDC","s":"_ZN5mongo12_GLOBAL_N_114RunCommandImpl3runEv","C":"mongo::(anonymous namespace)::RunCommandImpl::run()","s+":"13C"},{"a":"560998567BB4","b":"5609963F8000","o":"216FBB4","s":"ZN5mongo12_GLOBAL_N_119ExecCommandDatabase12_commandExecEv","C":"mongo::(anonymous namespace)::ExecCommandDatabase::commandExec()","s+":"1E4"},{"a":"56099856C85F","b":"5609963F8000","o":"217485F","s":"_ZZN5mongo12_GLOBAL_N_114executeCommandESt10shared_ptrINS0_13HandleRequest16ExecutionContextEEENUlvE0_clEv","C":"mongo::(anonymous namespace)::executeCommand(std::shared_ptr):: {lambda()#2}::operator()()","s+":"43F"},{"a":"56099856E450","b":"5609963F8000","o":"2176450","s":"ZZN5mongo15unique_functionIFvPNS_14future_details15SharedStateBaseEEE8makeImplIZNS1_10FutureImplINS1_8FakeVoidEE16makeContinuationIvZZNOS9_4thenIZNS_12_GLOBAL_N_114executeCommandESt10shared_ptrINSC_13HandleRequest16ExecutionContextEEEUlvE0_EEDaOT_ENKUlvE1_clEvEUlPNS1_15SharedStateImplIS8_EESN_E_EENS7_ISI_EEOT0_EUlS3_E_EEDaSJ_EN12SpecificImpl4callEOS3_","C":"mongo::unique_function::makeImpl::makeContinuation::then)::{lambda()#2} >(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1}>(mongo::future_details::FutureImpl::makeContinuation::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}>(mongo::future_details::FutureImpl::then)::{lambda()#2}>(mongo::(anonymous namespace)::executeCommand(std::shared_ptr)::{lambda()#2}&&) &&::{lambda()#3}::operator()() const::{lambda(mongo::future_details::SharedStateImpl, mongo::future_details::SharedStateImpl)#1}&&)::{lambda(mongo::future_details::SharedStateBase*)#1})::SpecificImpl::call(mongo::future_details::SharedStateBase*&&)","s+":"80"},{"a":"56099851EEB7","b":"5609963F8000","o":"2126EB7","s":"ZN5mongo14future_details15SharedStateBase20transitionToFinishedEv","C":"mongo::future_details::SharedStateBase::transitionToFinished()","s+":"147"},{"a":"5609985723B2","b":"5609963F8000","o":"217A3B2","s":"_ZN5mongo14future_details10FutureImplINS0_8FakeVoidEE11generalImplIZNOS3_17propagateResultToEPNS0_15SharedStateImplIS2_EEEUlOS2_E_ZNOS3_17propagateResultToES7_EUlONS_6StatusEE0_ZNOS3_17propagateResultToES7_EUlvE1_EEDaOT_OT0_OT1","C":"auto mongo::future_details::FutureImpl::generalImpl::propagateResultTo(mongo::future_details::SharedStateImpl*) &&:: {lambda(mongo::future_details::FakeVoid&&)#1} , mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda(mongo::Status&&)#2}, mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda()#3}>(mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&::{lambda(mongo::future_details::FakeVoid&&)#1}&&, mongo::future_details::FutureImpl::propagateResultTo(mongo::future_details::SharedStateImpl) &&:: JIRAUSER1271630 commented on Thu, 9 Mar 2023 13:51:08 +0000: When looking through our systemd logs, it looks like the mongos router actually exited once by itself about 2 minutes and 30 seconds before the crash above: Mar 9 08:19:30 appserver systemd[1]: mongos.service: Main process exited, code=exited, status=14/n/a Mar 9 08:22:00 appserver systemd[1]: mongos.service: Main process exited, code=killed, status=11/SEGV
Do not know how to reproduce. It happened randomly.
Click on a version to see all relevant bugs
MongoDB Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.