"docker restart on-failure" doesn't

Open Source Projects DockerEngine
1

I’m running a Docker container like so:

#!/bin/sh
exec docker run --name oops-it-leaks-memory \
  --log-driver gelf --log-opt gelf-address=udp://... \
  --dns ... -v ... -e ... \
  --restart on-failure \
  oops-it-leaks-memory

The single process in the container has some sort of memory leak in it that I haven’t diagnosed, and eventually the kernel OOM killer takes down that process. When it does, the restart policy doesn’t seem to apply. Any hints?

[If I was going to guess at something, it would be that, when the container exited, in spite of the restart policy, the docker run command exited too, and if I said docker run -d --restart on-failure it would work? I’d find that slightly surprising though.]

2

So the other part of this that turns out to be relevant is that this (foreground) docker run command is getting launched from a systemd unit file like so

[Service]
ExecStartPre=-docker kill oops-it-leaks-memory
ExecStartPre=-docker rm oops-it-leaks-memory
ExecStart=/usr/local/bin/start-oops-it-leaks-memory
ExecStop=docker stop oops-it-leaks-memory

Also, if you docker run --restart ... a container in the foreground, and it exits, the docker run command returns, and Docker restarts the container in the background.

So, the connection between all of this is that systemd sees the “start” script (that ended in exec docker run) exit, so it decides the service is stopped, and runs that ExecStop command. And now Docker has gotten an explicit docker stop so it doesn’t restart the service. The answer, in turn, is to tell systemd and not Docker to restart the container (with a nearly identical Restart=on-failure line in the unit file).

(I don’t think I can make systemd tolerate docker run -d without trying to write up a patch to include Docker in systemd; none of the service options seem to quite match. I can easily enough describe how primitives like “start a service” and “stop a service” and “check if the service is still running” map to Docker commands but don’t see an easy way to wire that all through.)