Info

We have noticed that mongodb community version 6.0.17 with openssl version openssl-1:3.2.2-6.el9 is causing huge cpu load spikes and keeps on accruing.Can easily be reproduced by installing the specific versions and sending some traffic. Edit by Technical Services: This issue is also observed in MongoDB Enterprise versions as well (7.0.14). Edit by Kubernetes Hosted: This issue is observed in every MongoDB Enterprise version running on RHEL 9.5 (or container images based on ubi9.5) where openssl is at 3.2.2 (CLOUDP-286015).

Top User Comments

JIRAUSER1262924 commented on Mon, 6 Jan 2025 22:28:44 +0000: Hey padeshmu@redhat.com , can you please try out this community build: ** link I believe the one provided may have been missing a necessary patch. As per previous test builds, this is provided SOLELY as a test mechanism to demonstrate that the fix works as expected. Please let me know how this goes. Cheers! JIRAUSER1281331 commented on Mon, 6 Jan 2025 15:48:11 +0000: Hello Team, We have tested the patched version against the openssl3.2.2 and issue is still persistent, Here are the detailed steps we performed - [root@xxx-xx-01 bin]# ./mongod -f /etc/mongod.conf {"t":\{"$date":"2025-01-06T15:30:36.033Z"} ,"s":"I",  "c":"CONTROL",  "id":5760901, "ctx":"-","msg":"Applied --setParameter options","attr":{"serverParameters":{"ocspEnabled":{"default":true,"value":false}}}} about to fork child process, waiting until server is ready for connections. forked process: 3614788 child process started successfully, parent exiting [root@xxx-xx-01 bin]# ./mongod --version db version v6.0.17-patch-6769aedcb3c2840007ca491d Build Info: { "version": "6.0.17-patch-6769aedcb3c2840007ca491d", "gitVersion": "1b0ca02043c6d35d5cfdc91e21fc00a05d901539", "openSSLVersion": "OpenSSL 3.2.2 4 Jun 2024", "modules": [], "allocator": "tcmalloc", "environment": { "distmod": "rhel90", "distarch": "x86_64", "target_arch": "x86_64" } } CPU load - top - 15:37:50 up 17 days, 17:55,  3 users,  load average: 37.38, 17.41, 7.02 Tasks: 312 total, 1 running, 311 sleeping, 0 stopped, 0 zombie %Cpu0  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st %Cpu1  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st %Cpu2  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st %Cpu3  : 99.0 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.3 si,  0.0 st %Cpu4  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st %Cpu5  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st %Cpu6  : 99.7 us,  0.3 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st %Cpu7  : 99.7 us,  0.0 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.3 hi,  0.0 si,  0.0 st MiB Mem :  31839.0 total,  10841.5 free, 2105.7 used,  19238.9 buff/cache MiB Swap: 0.0 total, 0.0 free, 0.0 used.  29733.4 avail Mem PID USER PR  NI VIRT RES SHR S  %CPU  %MEM TIME+ COMMAND 3614788 root 20 0 3997824 868120  49792 S 797.0 2.7  33:27.47 mongod mongo shell version - [root@xxx-xx-01 bin]# mongosh --tls --tlsAllowInvalidCertificates Current Mongosh Log ID: 677bf8f45df6c4a410964032 Connecting to: mongodb://127.0.0.1:27017/?directConnection=true&serverSelectionTimeoutMS=2000&tls=true&tlsAllowInvalidCertificates=true&appName=mongosh+2.3.1 Using MongoDB: 6.0.17-patch-6769aedcb3c2840007ca491d Using Mongosh: 2.3.1 mongosh 2.3.7 is available for download: https://www.mongodb.com/try/download/shell For mongosh info see: https://www.mongodb.com/docs/mongodb-shell/ xxx [direct: primary] test> rs.status() Kindly let us know if anything else you need from our end. JIRAUSER1262924 commented on Sat, 4 Jan 2025 00:05:40 +0000: Reposting from gabriel.marks@mongodb.com comment which was not visible externally : Hi aproddut@redhat.com, here is a patched release for community: Link. Same as before, this should be used as a test mechanism to ensure we are fixing your issue here. Please follow up on this case once you've tested it out. Thanks. aproddut@redhat.com commented on Mon, 23 Dec 2024 17:23:54 +0000: I see this seems to be for an enterprise version. Can a binary release be provided for the community version which we are using ? JIRAUSER1260495 commented on Mon, 23 Dec 2024 16:30:28 +0000: Hello aproddut@redhat.com can you please follow this tutorial (https://www.mongodb.com/docs/v6.2/tutorial/monitor-with-snmp/) to get SNMP setup? In your case the relevant SNMP files are located in `dist-test/snmp`. Please let me know if you have any trouble linking these files. aproddut@redhat.com commented on Mon, 23 Dec 2024 14:06:33 +0000: Are there any new library requirements with the binary version .. getting the following error while trying to get the version info [root@usermongo-qa-01 bin]# pwd /root/dist-test/bin [root@usermongo-qa-01 bin]# ./mongod --version ./mongod: error while loading shared libraries: libnetsnmpmibs.so.40: cannot open shared object file: No such file or directory aproddut@redhat.com commented on Sun, 22 Dec 2024 01:05:25 +0000: I will try to get this tested on monday .. JIRAUSER1274749 commented on Fri, 20 Dec 2024 20:14:54 +0000: Hi aproddut@redhat.com! We have prepped a binary release of a potential fix for you to test with. This is provided SOLELY as a test mechanism to demonstrate that the fix works as expected. Please update this case if you've any issues and we will address as soon as possible. Thanks.  Jerry Gilyeat Technical Account Manager aproddut@redhat.com commented on Mon, 16 Dec 2024 15:44:01 +0000: Hey adam.rayner@mongodb.com There are about nearly 250 connections to the DB currently in our non-prod environment .. most of them connecting via java driver .. info from the logs _ {"t":\{"$date":"2024-12-16T15:31:47.038+00:00"} ,"s":"I",  "c":"NETWORK",  "id":51800, "ctx":"conn154870","msg":"client metadata","attr":{"remote":"10.xx.xx.xx:58462","client":"conn154870","negotiatedCompressors":[],"doc":{"driver": {"name":"mongo-java-driver|legacy","version":"4.6.1"} ,"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.14.0-284.82.1.el9_2.x86_64"},"platform":"Java/Red Hat, Inc./17.0.13+11-LTS"}}}_ There are few which are connecting using _ {"t":\{"$date":"2024-12-16T15:31:48.993+00:00"} ,"s":"I",  "c":"NETWORK",  "id":51800, "ctx":"conn154872","msg":"client metadata","attr":{"remote":"10.xx.xx.xx:34742","client":"conn154872","negotiatedCompressors":[],"doc":{"driver": {"name":"mongo-java-driver|sync","version":"5.1.0"} ,"os":{"type":"Linux","name":"Linux","architecture":"amd64","version":"5.14.0-284.82.1.el9_2.x86_64"},"platform":"Java/Red Hat, Inc./17.0.13+11-LTS","env":{"container": {"orchestrator":"kubernetes"} }}}}_ Attaching our mongod.conf file. We see the behavior in less than 5 to 10 min on a busy system.. and looks like it all dependent on the traffic to the DB .. Some of our other replicasets which are not very busy the load spikes up slowly in a day or two Very quick to reproduce this . upgrade the openssl rpms on the host and restart mongod service .. if the node is secondary, the load might take little longer to come up .. but we we make this host primary again .. the load shoots up in less than 5 to 10 min .. let me know if you need any more info JIRAUSER1262924 commented on Fri, 13 Dec 2024 20:31:06 +0000: Hi aproddut@redhat.com can you provide more detailed steps about the reproduction of this issue? In particular, MongoDB driver language and version used to send traffic, specific server configuration parameters, how many connections, and for how long until this behaviour is observed? Thanks.

Steps to Reproduce