Now that I started dabbling in rust, I find myself writing more and more multithreaded code, because rust really makes it as easy, as it advertises it. While that would be entire series of articles itself, I want to write about something else, but related first.
I wrote my first multithreaded web service not too recently, that spins up a HTTP server, and when handling requests, it will schedule long running jobs onto a separate thread. The idea is that the job takes so long to run that holding the HTTP connection open becomes infeasible, so the service just validates the request, and if it's fine, it sends a job to a background thread, and immediately sends back a 200 OK to the client. (Unfortunately, this also makes it impossible to send any meaningful response back to the client about the result of the background job, without saving the results for later, this is the same problem as using job queues).
I deployed my application, and wrote a small .service file for it, so that it can be managed by systemd. Everything was looking great, but after a few days I noticed that some jobs that are supposed to be handled by the background thread get "half done". Some artifacts like cache files are laying around, but the actual result of the job is nowhere. Looking at the logs, I saw that there seems to be a correlation between the application restarting, and the jobs failing. This was a hint that made me discover that when I stop/restart my service, systemd basically kills not just the process, but all the child processes as well.
This was quite alarming at first, because this would have meant that I have to do some signal handling that would be much more complicated than I expected, or was comfortable with at the moment, but it turns out, the systemd author(s) have already thought about this. Systemd comes with a configuration option for services, called KillMode, that controls how the service gets stopped on a systemctl restart/stop command, and it's very nicely documented in the systemd.kill man page.
KillMode=
Specifies how processes of this unit shall be killed. One of control-group, mixed, process, none.
If set to control-group, all remaining processes in the control group of this unit will be killed on unit stop (for services: after the stop command is executed, as configured with ExecStop=). If set to mixed, the SIGTERM signal (see below) is sent to the main process while the subsequent SIGKILL signal (see below) is sent to all remaining processes of the unit's control group.
Perfect. So the default, which is control-group basically sends a SIGTERM to everything. SIGTERM can be handled by the application, depending on what programming language and/or runtime environment you use, it might already handle it for you. For me, this basically shut down every thread immediately. If the application still does not exit after getting SIGTERM, systemd will eventually send a SIGKILL (which is basically the OS going "goodbye" on the application, and shooting it).
I just had to change this to mixed, which means that only the main process gets the SIGTERM signal, and then it can decide what to do with it, however, if it fails to exit, the SIGKILL will be sent to every thread, working as a safety net for cleaning up everything, if the application's own cleanup handling failed. With the help of a nice tokio article on graceful shutdown I managed to implement a cleanup procedure that waits for the jobs to complete, before shutting down.
There are also many other options and modes described in the systemd.kill man page, I recommend reading it.