Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
// code placeholder {"t":{"$date":"2023-01-11T08:32:59.046+00:00"},"s":"W", "c":"CONTROL", "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"ASSERT", "id":23079, "ctx":"initandlisten","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":156}}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"ASSERT", "id":23080, "ctx":"initandlisten","msg":"\n\n***aborting after invariant() failure\n\n"}{"t":{"$date":"2023-01-11T08:32:59.060+00:00"},"s":"F", "c":"CONTROL", "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} we have this issue upgrading 2 different mongo instances, 1 is a standalone and another replicaset with 3 nodes. FCV set to 5.0 We can't understand why it blocks.
JIRAUSER1272552 commented on Fri, 13 Jan 2023 15:10:03 +0000: Thank you Edwin for your investigation. We will wait for the fix to try the upgrade again. Best Regards, Alessandro JIRAUSER1257066 commented on Fri, 13 Jan 2023 14:56:05 +0000: Hi alessandro.lombardi@coreview.com, After investigating this invariant, the issue you're hitting is SERVER-71068, where a mongod on 6.0 with FCV set to 5.0 will fail to start up when the server is checking whether the index spec is valid. SERVER-71068 should fix this problem by not checking FCV during startup. I will close this as a duplicate of SERVER-71068, which you may watch for updates. Best, Edwin JIRAUSER1257066 commented on Thu, 12 Jan 2023 14:21:50 +0000: Thank you for uploading the log, I can confirm that I'm able to open the log file. I will pass this ticket on to the appropriate team to further investigate this invariant failure on startup. JIRAUSER1272552 commented on Thu, 12 Jan 2023 14:16:54 +0000: yes exactly, this is what I mean. I have uploaded now the log to the link you provide me, let me know if you can open the file. JIRAUSER1257066 commented on Thu, 12 Jan 2023 14:08:53 +0000: Hi alessandro.lombardi@coreview.com, Thank you for the swift follow up and the additional information! I presume when you say, "it is a loop of the messages posted above" that you mean that on every attempt to start the mongod on 6.0, the aborts when hitting this invariant. Since this invariant appears to be hit on startup, could you attach a single occurrence of the log from startup to when mongod hits the invariant failure? JIRAUSER1272552 commented on Thu, 12 Jan 2023 13:37:29 +0000: ok I try to upload the file to you but is a loop of the messages posted above. The 2 instances where we have the problem are: one standalone in docker one replicaset managed by cloud manager (where we have open a ticket to support currently in progress) the log shows the same messages. FCV is correcly set and the answer is ok { "featureCompatibilityVersion" : \{ "version" : "5.0" } , "ok" : 1 } we have another docker standalone where the upgrade was successful, same config of the other one. I believe that the problem could be on data but I don't know what to check. Rollback to 5.0.14 works, issue was also present upgrading to 6.x from 5.0.8. This mongo data are coming from old 4.x versions where we have not experienced any issue upgrading to 5.x JIRAUSER1257066 commented on Thu, 12 Jan 2023 13:13:28 +0000: The invariant seems to be coming from a feature flag check during startup recovery, where FCV does not seem to have been set yet. https://github.com/mongodb/mongo/blob/r6.0.3/src/mongo/db/server_options.h#L147-L158 JIRAUSER1257066 commented on Thu, 12 Jan 2023 13:09:46 +0000: Hi alessandro.lombardi@coreview.com, Thank you for submitting this incident. Would you please archive (tar or zip) the mongod.log files for this incident and upload them to this support uploader location? Files uploaded to this portal are visible only to MongoDB employees and are routinely deleted after some time. Kind regards, Edwin JIRAUSER1272552 commented on Thu, 12 Jan 2023 09:08:29 +0000: {"t":\{"$date":"2023-01-11T08:32:59.046+00:00"} ,"s":"W", "c":"CONTROL", "id":5123300, "ctx":"initandlisten","msg":"vm.max_map_count is too low","attr":{"currentValue":65530,"recommendedMinimum":1677720,"maxConns":838860},"tags":["startupWarnings"]} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"} ,"s":"F", "c":"ASSERT", "id":23079, "ctx":"initandlisten","msg":"Invariant failure","attr":{"expr":"isVersionInitialized()","file":"src/mongo/db/server_options.h","line":156}} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"} ,"s":"F", "c":"ASSERT", "id":23080, "ctx":"initandlisten","msg":"\n\n***aborting after invariant() failure\n\n"} {"t":\{"$date":"2023-01-11T08:32:59.060+00:00"} ,"s":"F", "c":"CONTROL", "id":6384300, "ctx":"initandlisten","msg":"Writing fatal message","attr":{"message":"Got signal: 6 (Aborted).\n"}} {"t":\{"$date":"2023-01-11T08:33:25.236+00:00"} ,"s":"I", "c":"CONTROL", "id":31380, "ctx":"initandlisten","msg":"BACKTRACE","attr":{"bt":{"backtrace":[ {"a":"55ACD2015094","b":"55ACCD247000","o":"4DCE094","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTraceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.362","C":"mongo::stack_trace_detail::(anonymous namespace)::printStackTraceImpl(mongo::stack_trace_detail::(anonymous namespace)::Options const&, mongo::StackTraceSink*) [clone .constprop.362]","s+":"1F4"} ,{"a":"55ACD20175D9","b":"55ACCD247000","o":"4DD05D9","s":"ZN5mongo15printStackTraceEv","C":"mongo::printStackTrace()","s+":"29"},{"a":"55ACD20112B6","b":"55ACCD247000","o":"4DCA2B6","s":"abruptQuit","s+":"66"},{"a":"7FE7D737F420","b":"7FE7D736B000","o":"14420","s":"funlockfile","s+":"60"},{"a":"7FE7D71BC00B","b":"7FE7D7179000","o":"4300B","s":"gsignal","s+":"CB"},{"a":"7FE7D719B859","b":"7FE7D7179000","o":"22859","s":"abort","s+":"12B"},{"a":"55ACCF13FDDA","b":"55ACCD247000","o":"1EF8DDA","s":"_ZN5mongo15invariantFailedEPKcS1_j","C":"mongo::invariantFailed(char const*, char const*, unsigned int)","s+":"F7"},{"a":"55ACCF1096D8","b":"55ACCD247000","o":"1EC26D8","s":"_ZNK5mongo11FeatureFlag9isEnabledERKNS_18ServerGlobalParams20FeatureCompatibilityE.cold.76","C":"mongo::FeatureFlag::isEnabled(mongo::ServerGlobalParams::FeatureCompatibility const&) const [clone .cold.76]","s+":"18"},{"a":"55ACD02471E7","b":"55ACCD247000","o":"30001E7","s":"_ZNK5mongo16IndexCatalogImpl9_isSpecOkEPNS_16OperationContextERKNS_13CollectionPtrERKNS_7BSONObjE","C":"mongo::IndexCatalogImpl::_isSpecOk(mongo::OperationContext*, mongo::CollectionPtr const&, mongo::BSONObj const&) const","s+":"8C7"},{"a":"55ACD024B174","b":"55ACCD247000","o":"3004174","s":"_ZN5mongo16IndexCatalogImpl16createIndexEntryEPNS_16OperationContextEPNS_10CollectionESt10unique_ptrINS_15IndexDescriptorESt14default_deleteIS6_EENS_21CreateIndexEntryFlagsE","C":"mongo::IndexCatalogImpl::createIndexEntry(mongo::OperationContext*, mongo::Collection*, std::unique_ptr >, mongo::CreateIndexEntryFlags)","s+":"84"},{"a":"55ACD024ADA8","b":"55ACCD247000","o":"3003DA8","s":"_ZN5mongo16IndexCatalogImpl4initEPNS_16OperationContextEPNS_10CollectionE","C":"mongo::IndexCatalogImpl::init(mongo::OperationContext*, mongo::Collection*)","s+":"848"},{"a":"55ACD021E2C5","b":"55ACCD247000","o":"2FD72C5","s":"_ZN5mongo14CollectionImpl4initEPNS_16OperationContextE","C":"mongo::CollectionImpl::init(mongo::OperationContext*)","s+":"3C5"},{"a":"55ACD02377A1","b":"55ACCD247000","o":"2FF07A1","s":"_ZN5mongo12DatabaseImpl4initEPNS_16OperationContextE","C":"mongo::DatabaseImpl::init(mongo::OperationContext*)","s+":"121"},{"a":"55ACD0231DAF","b":"55ACCD247000","o":"2FEADAF","s":"_ZN5mongo18DatabaseHolderImpl6openDbEPNS_16OperationContextERKNS_18TenantDatabaseNameEPb","C":"mongo::DatabaseHolderImpl::openDb(mongo::OperationContext*, mongo::TenantDatabaseName const&, bool*)","s+":"30F"},{"a":"55ACD07549C6","b":"55ACCD247000","o":"350D9C6","s":"_ZN5mongo9AutoGetDb14ensureDbExistsEPNS_16OperationContextE","C":"mongo::AutoGetDb::ensureDbExists(mongo::OperationContext*)","s+":"B6"},{"a":"55ACD042068E","b":"55ACCD247000","o":"31D968E","s":"_ZN5mongo27FeatureCompatibilityVersion39findFeatureCompatibilityVersionDocumentEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::findFeatureCompatibilityVersionDocument(mongo::OperationContext*)","s+":"20E"},{"a":"55ACD0420F4C","b":"55ACCD247000","o":"31D9F4C","s":"_ZN5mongo27FeatureCompatibilityVersion20initializeForStartupEPNS_16OperationContextE","C":"mongo::FeatureCompatibilityVersion::initializeForStartup(mongo::OperationContext*)","s+":"4C"},{"a":"55ACCF39A135","b":"55ACCD247000","o":"2153135","s":"_ZN5mongo12_GLOBAL_N_115startupRecoveryEPNS_16OperationContextEPNS_13StorageEngineENS3_17LastShutdownStateENS_16startup_recovery19StartupRecoveryModeE","C":"mongo::(anonymous namespace)::startupRecovery(mongo::OperationContext*, mongo::StorageEngine*, mongo::StorageEngine::LastShutdownState, mongo::startup_recovery::StartupRecoveryMode)","s+":"55"},{"a":"55ACCF39A8E9","b":"55ACCD247000","o":"21538E9","s":"ZN5mongo16startup_recovery25repairAndRecoverDatabasesEPNS_16OperationContextENS_13StorageEngine17LastShutdownStateE","C":"mongo::startup_recovery::repairAndRecoverDatabases(mongo::OperationContext*, mongo::StorageEngine::LastShutdownState)","s+":"269"},{"a":"55ACCF35DC91","b":"55ACCD247000","o":"2116C91","s":"_ZN5mongo12_GLOBAL_N_114_initAndListenEPNS_14ServiceContextEi.isra.2009","C":"mongo::(anonymous namespace)::initAndListen(mongo::ServiceContext*, int) [clone .isra.2009]","s+":"A81"},{"a":"55ACCF360EED","b":"55ACCD247000","o":"2119EED","s":"_ZN5mongo11mongod_mainEiPPc","C":"mongo::mongod_main(int, char**)","s+":"BDD"},{"a":"55ACCF150FCE","b":"55ACCD247000","o":"1F09FCE","s":"main","s+":"E"},{"a":"7FE7D719D083","b":"7FE7D7179000","o":"24083","s":"_libc_start_main","s+":"F3"},{"a":"55ACCF35B74E","b":"55ACCD247000","o":"211474E","s":"_start","s+":"2E"}],"processInfo":{"mongodbVersion":"6.0.3","gitVersion":"f803681c3ae19817d31958965850193de067c516","compiledModules":[],"uname": {"sysname":"Linux","release":"4.4.0-186-generic","version":"#216-Ubuntu SMP Wed Jul 1 05:34:05 UTC 2020","machine":"x86_64"} ,"somap":[\{"b":"55ACCD247000","elfType":3,"buildId":"1D29859BFD681BF978651663140EBA623DD9D6E7"},\{"b":"7FE7D736B000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"7B4536F41CDAA5888408E82D0836E33DCF436466"},\{"b":"7FE7D7179000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"1878E6B475720C7C51969E69AB2D276FAE6D1DEE"}]}}},"tags":[]}
upgrade from 5.0.14 to 6.0.x