Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
Version : 4.4.10 In Ubuntu 18 all of the sudden Mongodb Server stopped working I found permission of /var/lib/mongodb is 755 and in /var/lib/mongodb/diagnostic.data, there are metrics files generated in every 2 days and a metrics.interim file ``` {"t":\{"$date":"2021-12-25T15:03:55.428+09:00"} ,"s":"I", "c":"NETWORK", "id":22944, "ctx":"conn24","msg":"Connection ended","attr":{"remote":"127.0.0.1:45082","connectionId":24,"connectionCount":7}} 48609 {"t": {"$date":"2021-12-25T15:04:31.004+09:00"} ,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"terminate() called. An exception is active; attempting to gather more information"}} 48610 {"t": {"$date":"2021-12-25T15:04:31.004+09:00"} ,"s":"F", "c":"CONTROL", "id":4757800, "ctx":"ftdc","msg":"Writing fatal message","attr":{"message":"DBException::toString(): FileStreamFailed: Failed to write to interim file buffer for full-time diagnostic data captu re: /var/lib/mongodb/diagnostic.data/metrics.interim.temp\nActual exception type: mongo::error_details::ExceptionForImpl\n"}} 48611 {"t": {"$date":"2021-12-25T15:04:31.045+09:00"} ,"s":"I", "c":"CONTROL", "id":31431, "ctx":"ftdc","msg":"BACKTRACE: {bt}","attr":{"bt":{"backtrace":[ {"a":"5556DA3F484A","b":"5556D7620000","o":"2DD484A","s":"_ZN5mongo18stack_trace_detail12_GLOBAL__N_119printStackTr aceImplERKNS1_7OptionsEPNS_14StackTraceSinkE.constprop.606","s+":"1EA"} ,{"a":"5556DA3F62D9","b":"5556D7620000","o":"2DD62D9","s":"ZN5mongo15printStackTraceEv","s+":"29"},{"a":"5556DA3F34A6","b":"5556D7620000","o":"2DD34A6","s":"_ZN5mongo12_GLOBAL_N_111myTerminateE v","s+":"A6"},{"a":"5556DA583476","b":"5556D7620000","o":"2F63476","s":"ZN10cxxabiv111terminateEPFvvE","s+":"6"},{"a":"5556DA6174D9","b":"5556D7620000","o":"2FF74D9","s":"cxa_call_terminate","s+":"39"},{"a":"5556DA582E65","b":"5556D7620000","o":"2F62E65","s": "gxx_personality_v0","s+":"275"},{"a":"7F79D45CD573","b":"7F79D45BD000","o":"10573","s":"_Unwind_GetTextRelBase","s+":"1E03"},{"a":"7F79D45CDAD1","b":"7F79D45BD000","o":"10AD1","s":"_Unwind_RaiseException","s+":"2B1"},{"a":"5556DA5835D7","b":"5556D7620000","o":"2F 635D7","s":"cxa_throw","s+":"37"},{"a":"5556D854D900","b":"5556D7620000","o":"F2D900","s":"_ZN5mongo13error_details23throwExceptionForStatusERKNS_6StatusE","s+":"1B72"},{"a":"5556D8561B1D","b":"5556D7620000","o":"F41B1D","s":"_ZN5mongo21uassertedWithLocationERKNS 6StatusEPKcj","s+":"27B"},{"a":"5556D82BE9A5","b":"5556D7620000","o":"C9E9A5","s":"_ZN5mongo14FTDCController6doLoopEv.cold.395","s+":"2D"},{"a":"5556D8AED15C","b":"5556D7620000","o":"14CD15C","s":"_ZNSt6thread11_State_implINS_8_InvokerISt5tupleIJZN5mongo4stdx6thread C4IZNS3_14FTDCController5startEvEUlvE0_JELi0EEET_DpOT0_EUlvE_EEEEE6_M_runEv","s+":"5C"},{"a":"5556DA59EF3F","b":"5556D7620000","o":"2F7EF3F","s":"execute_native_thread_routine","s+":"F"},{"a":"7F79D43A56DB","b":"7F79D439E000","o":"76DB","s":"start_thread","s+":"DB"} ,{"a":"7F79D40CE71F","b":"7F79D3FAD000","o":"12171F","s":"clone","s+":"3F"}],"processInfo":{"mongodbVersion":"4.4.10","gitVersion":"58971da1ef93435a9f62bf4708a81713def6e88c","compiledModules":[],"uname": {"sysname":"Linux","release":"5.4.0-81-generic","version":"#91~ 18.04.1-Ubuntu SMP Fri Jul 23 13:36:29 UTC 2021","machine":"x86_64"} ,"somap":[{"b":"5556D7620000","elfType":3,"buildId":"706585359197300EBD464448C618DD26F2A4291F"},{"b":"7F79D45BD000","path":"/lib/x86_64-linux-gnu/libgcc_s.so.1","elfType":3,"buildId":"679F3AE11120EC 7C483BC9295345D836F5C104F7"},{"b":"7F79D439E000","path":"/lib/x86_64-linux-gnu/libpthread.so.0","elfType":3,"buildId":"68F36706EB2E6EEE4046C4FDCA2A19540B2F6113"},{"b":"7F79D3FAD000","path":"/lib/x86_64-linux-gnu/libc.so.6","elfType":3,"buildId":"CE450EB01A5E5ACC7CE7 B8C26 ```
dmitry.agranat commented on Sun, 16 Jan 2022 10:31:49 +0000: Hi kdaeyoub@gmail.com, given this is an indication of either running out of disk space of other disk-related issues (for example, quote limitations), I will go ahead and close this case. JIRAUSER1264129 commented on Thu, 30 Dec 2021 00:49:03 +0000: Oh Okay then I will check it. Thank you. dmitry.agranat commented on Wed, 29 Dec 2021 09:41:07 +0000: Both backtraces in the log are cut off in the middle, meaning the OS was unable to complete writing even a few more bytes. This is usually an indication of either running out of disk space of other disk-related issues (for example, quote limitations) JIRAUSER1264129 commented on Wed, 29 Dec 2021 01:21:25 +0000: I didn't follow up of out of disk space. But I guess it is not. Because I found this error a week ago and that time I just restarted mongod and I went well. From that time, several days later, I found this error again. metrics files were written after when I restarted mongod. If the root cause is out of disk space, I would find the error as soon as I restarted the mongod. But I went well for a few days. But It is only my opinion so let me upload mongod.log file first. Thank you for your service. dmitry.agranat commented on Tue, 28 Dec 2021 10:55:55 +0000: kdaeyoub@gmail.com I am getting "404: Not Found" when trying to access your log. Did you check that you didn't run out of disk space? JIRAUSER1264129 commented on Tue, 28 Dec 2021 01:43:37 +0000: Mongod.log is (https://raw.githubusercontent.com/DavidKimDY/uipa_as/log/mongod.log?token=AMDUG5NSZ4PU2G6EKML6W2TBZJVZI) dmitry.agranat commented on Mon, 27 Dec 2021 09:48:37 +0000: Thanks for the report kdaeyoub@gmail.com. The backtrace seems to be incomplete, could you provide: the actual log containing this backtrace and a few lines before? last metrics. file located under /var/lib/mongodb/diagnostic.data? Also, could you check that you didn't run out of disk space?
mongod.conf ``` Where and how to store data. storage: dbPath: /var/lib/mongodb journal: enabled: true # engine: # mmapv1: # wiredTiger:# where to write logging data. systemLog: destination: file logAppend: true path: /var/log/mongodb/mongod.log# network interfaces net: port: 27017 bindIp: 127.0.0.1# how the process runs processManagement: timeZoneInfo: /usr/share/zoneinfo```