I have a machine running ubuntu and a docker service with 3 containers
the problem is that when the machine shuts down , I want the docker container to receive a SIGTERM signal that can be caught by the containers and allow them to shut down gracefully.
when I do “docker stop $ID” the container shuts down gracefully as expected
but when powering off the machine (shutdown -h now") the docker service exits and sends a SIGKILL to the containers which I am unable to catch and handle.
does anyone know how to solve that?
I’ve been researching this as well today and have concluded the following:
There are a few things going on here. First, what do you use for an init system? Seeing as you’re running ubuntu, you are probably running systemd. And if you look at the systemd service file, there is no ExecStop command so the docker daemon simply does not support graceful shutdown. Unless someone has already updated the docker service file, I believe you will need to write an ExecStop script that does a docker ps, finds all running containers and then explicitly send a docker stop to them.
Now I have a second problem. We use systemd inside the containers as well Turns out, docker stop sends a SIGTERM to the init process of the container. After a while, it will follow that up with a SIGKILL. Unfortunately, systemd does not initiate shutdown on receiving a SIGTERM. The systemd documentation states that it is like a systemctl reexec call which simply is not what we want.
Fortunately, there is a docker kill command which can send arbitrary signals to the init process of a container. For systemd, this is SIGRTMIN+5. A container instance of systemd, upon receiving this signalwill initiate a graceful shutdown. So, what we want to run is a docker kill command instead of a docker stop command.
I know there has been discussions about running systemd as the init process of a docker container. Folks have recommended against it, but for various reasons, we prefer systemd. It is unfortunate that docker doesn’t support systemd out of the box. I have only run into two issues: On startup, systemd will not pass environment variables to services. This means that services will not inherit any explicitly defined environment variables for the container. The second is this graceful shutdown problem. The first issue is relatively easy to fix by adding a script snippet to services that copy the environment space of the init process.
If anyone knows of other issues and workarounds, please point them out. It would be good to handle this proactively.