How to handle server reboot when using docker-compose?

When using docker run for a single container, I understand that I can assign a restart policy to make sure the container is restarted by docker when the server reboots.

What is the equivalent practice when using docker-compose for all services defined by the docker-compose.yml?

If I stand up several services as containers using docker-compose, what is controlling the restart/reboot condition for everything as a whole and individually?

Have been wondering about this myself. Turns out after a few false starts Google is our friend, and that it’s simply part of the compose file syntax:

https://docs.docker.com/v1.7/compose/yml/#working-dir-entrypoint-user-hostname-domainname-mem-limit-privileged-restart-stdin-open-tty-cpu-shares-cpuset-read-only

Further on setting up a production environment:

  1. That does answer on a per service/container, sort of … that part of the documentation doesn’t give a clear example of where to use those parameters. It just lists the parameters as if they are floating around in the file “somewhere”.

and more importantly

  1. That doesn’t explain how to have the whole definition in the compose restart itself in the same way as when docker-compose up was run originally. There are things that defined only in the docker-compose.yml file. How are those handled?

For one example:

What is controlling the depends_on references to make sure the services/containers that are depended on start first and then the ones that depend on start after but not before - as is defined in the docker-compose.yml but is not defined for the container on its own.

On 1) I would expect the restart: to be nested under the service/container you want to set it for. So the example given in the depends_on section of the compose reference could be:

version: '2'
services:
  web:
    build: .
    restart: always         # <--- my addition 
    depends_on:
      - db
      - redis
  redis:
    image: redis
  db:
    image: postgres

Does that not work for you if you test out a similar addition in your environment?

Regarding 2) I would simply expect declarations as depends_on: to work just as it did before you add restart: So in the example given above a natural consequence after a restart would be that the dependent services db and redis will be started before web. Something similar should implicitly be true for volumes and networks your services refer to.

Sorry to necro this thread, but I needed an answer to this very question.

After trying this, I found I had to add restart: always to the redis: section as well to make sure it came up. With just depends_on: redis and restart: always in the web: section, the apache container would restart after a reboot, but redis wouldn’t.

No worries. I was wondering where to put that as wall. If I understand you correctly, you have to put “restart: always” in each service that you want to start on system boot?

I wasn’t sure if you could make a single “global” entry in the file that would applied to all of the services or not.

For example, the top of my file looks like this before I add restart: always to it


version: “3.6”
services:
web:
…web parameters
database:
…database parameters

Then with the restart: always parameter


version: “3.6”
restart: always ← here at top level?
services:
restart: always ← or here at services level ?
web:
…web parameters
restart: always ← or perhaps in each service, here?

database:
…database parameters
restart: always ← or perhaps in each service, here?

With docker-compose, the “restart:” node is a direct child node of a specific service, a sibling of “image:”
For Swarm stack deploymets, the “restart_policy:” node is a child node of “deploy:”, which itself is a sibling of “image:”

The problem with the restart policy at the service level is that docker itself is unaware of the depends_on between services that only appears in the docker-compose.yml file. docker-compose up will start services in dependency order and can wait on services to be healthy before starting their dependent services. But on reboot, every service that is marked for restart will be immediately started without waiting for their dependency services to be healthy first.
Is there a way to address that? It seems incomplete to have depends_on functionality if it only works in an explicit up call but not on reboot.

This is exactly what I am looking for. Is there any solution for this yet?

Hey, more than a year later, is there an answer to this? :sweat_smile:
I’m havin the exact same problem, I’m trying to start two containers after one another importing variables via the docker compose file and I need to restart them after a reboot/power outage…

Greetings Luis

Docker Compose is a kind of client for Docker. Features that Docker doesn’t support are used for developement and in some cases to make sure the first started container can initialize data which will be mounted by another container as well. So when you first start the container, the order of the containers are more important. I never used the depends_on feature with healthchecks as the applications should handle when a dependency is not available. If the app fails because it can’t access an endpoint and the container restarts, it can try again. Or endpoint checking can be added in a loop in the entrypoint before the application starts and run until the endpoint is ready. This works regardless of how where your containers are running (Kubernetes, Swarm, Compose).

There is more subtle problem. Consider you have two containers A and B:

A is a regular container.
B is a container with --network=container:A (in compose it would be network_mode: service:A).

Docker starts A and B in random order. If B happens to start first, it will fail with “cannot join network of a non running container:A” and compeltely ignore --restart=always because docker considers this as a configuration error.
This is quite a pain as currently there is no way to resolve this, except having some script that will check for such condition.

That’s a good point. Thanks for commenting. I don’t remember if I ever needed sharing the network between containers or anything else in production where auto start was needed at reboot. I used it in CI/CD pipelines and to connect to an existing container when running a scheduled job container or running one manually so I never actually saw that error message, but you are right.

In this case I would probably try a Systemd service that just executes the docker compose up command. You can probably find existing guides to do tha.

As far as I know, even Docker Swarm doesn’t support sharing network namespaces, but you can ask for supporting this case with the Docker Engine in the roadmap, so then it can be supported in Docker Compose as well.

If it is not possible in Docker Engine (using docker run commands), Docker Compose cannot help as a client application.

It does. It requires service:<servicename> instead of container:<containername>.

Docker compose supports the network namespace sharing, but Docker Swarm says network_mode is ignored as it is not supported. The quoted part of my message indeed looks like I was writing about the namespace sharing, but I just wanted to note that although Docker Swarm would be something I would try when something is not working in Compose, but it doesn’t seem to support the parameter, so it couldn’t be used instead of compose to handle shared network namespaces when starting containers in a stack at boot.

Did I miss something, or I just confused you with my previous message? :slight_smile:

I must have remembered it wrong :slight_smile: It a makes sense that it’s not supported. How should a container on another node use the same network namespace like a container on a different node…

I get that depends_on is a Compose-only feature that only has an effect when the stack is initially brought up and so Engine does nothing on reboot, but this seems a lot like a “shipping the org chart” shaped design gap that leaves the user needing to rely on an external dependency (systemd) to work around it.

Even if you fall back to using systemd to start up the stack, it seems like then you’re left with more bad choices. Doesn’t that leave you also unable to use other common Engine features like restart: unless-stopped to handle automatically restarting containers? How else would you have that engine restart policy apply normally but not after a reboot (until the systemd unit starts the stack)?

It isn’t reliable to say that the systemd unit should stop the stack before reboot. Clean startup shouldn’t rely on a previous clean teardown…systems can panic, lose power, who knows what. But then I’m not sure what other opportunity you’d have before engine startup following the reboot, which would then apply that restart policy without depends_on. The only reliable way I can think of to do this would be to move your stack’s restart policy to systemd too.

Having restart policy not linked to depends_on seems like a design problem. It makes depends_on more of a trap to avoid than a useful feature. In what real environment is a user going to want to start a stack where services have startup order dependencies, but where that dependency should not apply on every container start?

I had to read my comments again, because I didn’t remember, but I think I did not say that compose or Docker CE could not be improved. Features could be asked on the roadmap too. That is why I shared the link to in in my post.

You are right, that the current implementation of depends_on is mainly useful during develeopment for an interactive user, but Compose itself cannot do much more. So what I recommended was only for the current capabilities of Docker CE and Docker Compose until someone asks for new features to support dependencies during reboot in Docker CE or at least detects which container uses another container’s kernel namespaces and starts those first that don’t rely on any other.

Since it is something I didn’t need, I did not ask for this feature. If anyone does, please share the ticket link here as well so other users can go and support the feature request.