Launching soon: The world's first vendor agnostic bug scrubLearn more & join waitlist
OPERATIONAL DEFECT DATABASE
...
...
When starting mongod as a systemd service, the process forks before mongod has opened the socket and starts listening for connections. This can cause dependent services to fail to start, as systemd considers mongod started once the process has forked.
xgen-internal-githook commented on Tue, 5 Dec 2017 17:10:26 +0000: Author: {'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com', 'name': 'Mathew Robinson'} Message: SERVER-31225 Add Type=forking to systemd service files Branch: v3.6 https://github.com/mongodb/mongo/commit/7aa806063d4279a728a65fc9202a3928e36cd6f1 xgen-internal-githook commented on Thu, 30 Nov 2017 20:13:38 +0000: Author: {'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'} Message: SERVER-31225 Add Type=forking to systemd service files Branch: v3.4 https://github.com/mongodb/mongo/commit/42e14f07cb17a5619e54354de471a9b5d4f6e5fa xgen-internal-githook commented on Thu, 30 Nov 2017 19:16:11 +0000: Author: {'name': 'Mathew Robinson', 'username': 'chasinglogic', 'email': 'chasinglogic@gmail.com'} Message: SERVER-31225 Add Type=forking to systemd service files Branch: master https://github.com/mongodb/mongo/commit/3607059cd791b273ccaefd879d765dcd9365ebd6 mathew.robinson commented on Wed, 29 Nov 2017 23:08:20 +0000: I was able to reproduce using the systemd files from our packages: [root@ip-10-30-199-253 system]# systemctl start mongod && mongo MongoDB shell version v3.4.10-40-g2e9c0d0 connecting to: mongodb://127.0.0.1:27017 2017-11-29T23:07:03.452+0000 W NETWORK [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: Connection refused 2017-11-29T23:07:03.453+0000 E QUERY [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed : connect@src/mongo/shell/mongo.js:240:13 @(connect):1:6 exception: connect failed [root@ip-10-30-199-253 system]# With Type=forking: [root@ip-10-30-199-253 system]# systemctl start mongod && mongo MongoDB shell version v3.4.10-40-g2e9c0d0 connecting to: mongodb://127.0.0.1:27017 MongoDB server version: 3.4.10-40-g2e9c0d0 Server has startup warnings: 2017-11-29T23:07:38.472+0000 I STORAGE [initandlisten] 2017-11-29T23:07:38.472+0000 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine 2017-11-29T23:07:38.472+0000 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem 2017-11-29T23:07:46.919+0000 I CONTROL [initandlisten] 2017-11-29T23:07:46.919+0000 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database. 2017-11-29T23:07:46.919+0000 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted. 2017-11-29T23:07:46.919+0000 I CONTROL [initandlisten] MongoDB Enterprise > mark.agarunov commented on Fri, 6 Oct 2017 16:51:44 +0000: ernie.hershey, I confirmed that mongod does start listening before forking. This should more or less match the behavior of systemd with Type=forking: [vagrant@localhost ~]$ mongod --logpath log --dbpath data --fork && testsocket about to fork child process, waiting until server is ready for connections. forked process: 1978 child process started successfully, parent exiting Listening I believe that the Type=forking will fix this, as without it systemd will treat mongod as Type=simple: If set to simple (the default if neither Type= nor BusName=, but ExecStart= are specified), it is expected that the process configured with ExecStart= is the main process of the service. In this mode, if the process offers functionality to other processes on the system, its communication channels should be installed before the daemon is started up (e.g. sockets set up by systemd, via socket activation), as systemd will immediately proceed starting follow-up units. I reproduced this by launching mongod in a subshell: [vagrant@localhost ~]$ mongod --logpath log --dbpath data --fork & testsocket [1] 2006 about to fork child process, waiting until server is ready for connections. Ncat: Connection refused. Not listening [vagrant@localhost ~]$ forked process: 2010 child process started successfully, parent exiting which matches the behavior without Type=forking. ernie.hershey@10gen.com commented on Fri, 6 Oct 2017 16:28:59 +0000: henrik.edin are you certain that doing that will make Mark's reproduction case succeed? I think it will still fork early. henrik.edin commented on Mon, 2 Oct 2017 21:51:47 +0000: I've been investigating this with mark.agarunov and it seems that the core issue is that the default mongod.service file doesn't have Type=forking By default systemd is using a 'simple' type where the process is forked automatically at start which is what causing this issue. While looking into this it also seems like the default debian mongod.service and mongod.conf doesn't set fork: true in the processManagement block.
Create a systemd service to test if the socket is listening: [Unit] Type=oneshot Description=TestService After=mongod.service Wants=mongod.service [Service] ExecStart=/usr/bin/testsocket Restart=on-abort [Install] WantedBy=multi-user.target With the following script (/usr/bin/testsocket) #!/bin/sh /bin/nc localhost 27017 /dev/null && echo "Listening" || echo "Not listening" Stop mongod, and start the service which should start mongod first: [root@localhost ~]# systemctl stop mongod [root@localhost ~]# ps aux|egrep '[m]ongo' [root@localhost ~]# systemctl start testservice [root@localhost ~]# systemctl status testservice ● testservice.service - TestService Sep 21 19:49:02 localhost.localdomain systemd[1]: Started TestService. Sep 21 19:49:02 localhost.localdomain systemd[1]: Starting TestService... Sep 21 19:49:02 localhost.localdomain testsocket[3830]: Ncat: Connection refused. Sep 21 19:49:02 localhost.localdomain testsocket[3830]: Not listening Add a 1 second sleep via ExecStartPre in the testservice unit file, or to the beginning of /usr/bin/testsocket and mongod is listening by the time it runs: [root@localhost ~]# systemctl stop mongod [root@localhost ~]# ps aux|egrep '[m]ongo' [root@localhost ~]# systemctl start testservice [root@localhost ~]# systemctl status testservice ● testservice.service - TestService Sep 21 19:52:13 localhost.localdomain systemd[1]: Started TestService. Sep 21 19:52:13 localhost.localdomain systemd[1]: Starting TestService... Sep 21 19:52:14 localhost.localdomain testsocket[3882]: Listening