OPERATIONAL DEFECT DATABASE
...
...
I am running the offical mongo docker image, which has mongo version 8.2.1. This image runs fine on at least one host, but fails with an assertion failure on another. I'm not sure why it would matter, but the host it fails on is not connected to the external internet, just intranet. I am attempting to debug by looking and things in a shell via `docker run it --privileged -entrypoint bash mongo` The error, which happens after a simple `mongod --help` is: {"t":{"$date":"2025-10-29T18:56:46.031+00:00"},"s":"F", "c":"ASSERT", "id":23089, "ctx":"main","msg":"Fatal assertion","attr":{"msgid":40379,"location":"src/mongo/crypto/sha_block_openssl.cpp:163:5:void mongo::{anonymous}::computeHashImpl(const EVP_MD*, std::initializer_list, HashType*) [with HashType = mongo::MakeArrayType; EVP_MD = evp_md_st]"}} {"t":{"$date":"2025-10-29T18:56:46.032+00:00"},"s":"F", "c":"ASSERT", "id":23090, "ctx":"main","msg":"\n\n***aborting after fassert() failure\n\n"} I do not see any other indications of an error `/var/log` or `dmesg` output. I attempted to run with `-vvvvv` but the output is unchanged. Any insights or debugging suggestions would be very welcome! I'm leaving the priority at the default P1 – it's a blocker for me, but I don't think this error is common. I do see that it cropped up in 2024: https://www.mongodb.com/community/forums/t/fatal-assertion-possibly-related-to-openssl/303821
JIRAUSER1288532 commented on Wed, 29 Oct 2025 23:34:26 +0000: Verified, the following Dockerfile solves the issue, though that last part is heavy-handed: FROM mongo:latest AS build RUN apt-get update && apt-get install -y build-essential perl git ca-certificates RUN git clone --branch openssl-3.0.13 --depth 1 https://github.com/openssl/openssl.git RUN cd openssl \ && ./config fips enable-fips \ && make -j && make install FROM mongo:latest RUN rm -rf /usr/local COPY --from=build /usr/local /usr/local ENV LD_LIBRARY_PATH=/usr/local/lib64 Thanks again, I was a bit too quick to create an issue here, but maybe this will be helpful. JIRAUSER1265262 commented on Wed, 29 Oct 2025 22:50:37 +0000: Thanks a bunch. Your input will help when we get back to this. I am wondering if we can get a more actionable error message, or if this needs some specific guidance in our production notes. We do have https://www.mongodb.com/docs/manual/tutorial/configure-fips/ for setting up FIPS with mongodb JIRAUSER1288532 commented on Wed, 29 Oct 2025 22:44:51 +0000: It appears that the host that doesn't work has FIPS enabled and that somehow flows into the container. The openssl installed in the mongo image does not have a FIPS provider. I will be experimenting making a new FIPS-enabled image to verify that is the problem. Thank you very much for assisting, I know this is more of a support issue than a bug. Your help is very much appreciated – hopefully it can improve something on your end also. JIRAUSER1265262 commented on Wed, 29 Oct 2025 22:40:04 +0000: Edit - I would be interested in getting that same output on the node that is working (not crashing) as well just to compare. Are there any host level differences between the two? What environments are they running in? (os, cpu architecture) I am going to assign this to another team to double check if this assertion is due to a host level misconfiguration, and if so determine what mongod expects to start up without this error. JIRAUSER1288532 commented on Wed, 29 Oct 2025 21:40:51 +0000: Many thanks for the quick response! The objdumps seemed to have reasonably matching symbols, but I uploaded them to that portal (there are no "good" versions as explained below). One of the very confusing issues is that this exact set of software (being a docker image) works when run on a different host. There seems to be something about the host system that is causing a problem. I'm imagining a kernelspace crypto interface or something similar. As such, the problem is probably not necessarily with mongod, though it might serve it well to fail more gracefully in this case. If you have any ideas for troubleshooting along those lines, please let me know. When I can, I will be creating a new mongo image that contains some tools like strace to look deeper into what's happening. Also, there are no logs and the db directory is not populated. mongod appears to crash very early before doing any of that. As I mentioned, simply running with `--help` is enough to see the crash. I will update this issue when I can get strace output. Edit to add that no, there is no stack trace. JIRAUSER1265262 commented on Wed, 29 Oct 2025 21:07:52 +0000: Thanks for your report. If a stack trace is printed along side it, can we get the full stack trace? It would also be helpful to get some diagnostic data from the node that works versus the one that doesn't. I wonder if there is an SSL configuration difference between the hosts that can explain it. Next steps: First please verify you are following our production notes. Specifically getting output from this section on a good node vs a bad node would help narrow down the issue on each host: objdump -T /mongod | grep " SSL_" objdump -T /mongod | grep " CRYPTO_" compared to objdump -T /libssl.so.1* objdump -T /libcrypto.so.1* Along with that, to look into this further, I've created a secure upload portal for you. Files uploaded to this portal are hosted on Box, are visible only to MongoDB employees, and are routinely deleted after some time. For each node in the replica set spanning a time period that includes the incident, would you please archive (tar or zip) and upload to that link: the mongod logs the $dbpath/diagnostic.data directory (the contents are described here)
$ docker run -it --entrypoint bash mongo:8.2.1 # mongod --help But it only happens on some hosts.
Click on a version to see all relevant bugs
MongoDB Integration
Learn more about where this data comes from
Bug Scrub Advisor
Streamline upgrades with automated vendor bug scrubs
BugZero Enterprise
Wish you caught this bug sooner? Get proactive today.
