Docker compose project does not start properly upon reboot/service restart

Docker Engine Compose
, , , ,
1

Hey,

I have a setup, where I have a docker-compose.yml file that defines some relationships between my containers, mainly dependency on other containers.

My particular issue is with logging: failed to create task for container: failed to initialize logging driver: dial tcp 127.0.0.1:5514: connect: connection refused.

I have a syslog container with a healthcheck to make sure it starts before I start other containers, but it seems like when I reboot, the depends_on criteria is not handled properly. The syslog container binds to a port on the local host, so that other containers can use it for logging via 127.0.0.1. I made sure to specify condition: service_healthy for other containers that depend on syslog.

Excerpt from my docker-compose.yml:

# vim:syntax=yaml

services:
  syslog:
    container_name: "syslog"
    image: "syslog:develop"
    build: /opt/syslog
    restart: always
    environment:
      TZ: "Europe/Bratislava"
    volumes:
      - "/opt/syslog/:/etc/syslog-ng/conf.d/:ro"
    ports:
      - "127.0.0.1:5514:514"
    logging:
      driver: json-file
      options:
        max-size: "10m"
        max-file: "20"

  traefik:
    restart: always
    depends_on:
      syslog:
        condition: service_healthy
    logging:
      driver: syslog
      options:
        syslog-address: "tcp://127.0.0.1:5514"
        syslog-facility: daemon
        tag: "{{.Name}}"
    container_name: "traefik"
    image: traefik:latest
    environment:
      TZ: "Europe/Bratislava"
    volumes:
      - "/opt/traefik/:/etc/traefik/:ro"
      - "/opt/certs:/opt/certs:ro"
      - /var/run/docker.sock:/var/run/docker.sock:ro
    ports:
      - "80:80"
      - "443:443"
      - "127.0.0.1:9090:9090"
    networks:
      - traefik_network

networks:
  traefik_network:

Syslog container:

FROM alpine:3.19

RUN apk add --no-cache \
    glib \
    pcre \
    eventlog \
    openssl \
    tzdata \
    syslog-ng

RUN mkdir -p /etc/syslog-ng/conf.d /var/run/syslog-ng

ADD syslog-ng.conf /etc/syslog-ng
VOLUME ["/var/run/syslog-ng", "/etc/syslog-ng/conf.d"]

EXPOSE 514/udp 514/tcp 601/tcp 6514/tcp

ENTRYPOINT ["/usr/sbin/syslog-ng"]
CMD ["-F", "-f", "/etc/syslog-ng/syslog-ng.conf"]

HEALTHCHECK --interval=5s --timeout=1s --start-period=3s --retries=3 CMD syslog-ng-ctl stats || exit 1

If I use docker compose up -d everything starts fine and works. But when I reboot the machine, or just systemctl restart docker, the syslog container starts fine and says it is Up 17 minutes (healthy), but all other containers become Exited (128) 17 minutes ago and never attempt to restart, despite the policy of restart: always.

It seems to me like the dependency check is not performed when the docker daemon starts, or it does not apply outside of manually running a service with docker compose up -d which is a terrible design IMHO.

How am I supposed to work around this issue? I was hoping that by defining all the conditions in the docker-compose.yml file would ensure that syslog is started before all other containers that depend on it, and the daemon would wait until the service is healthy before bringing up other containers that depend on it. It looks like when the docker daemon restarts, it starts all containers at the same time, making the containers that depend on syslog exit and never recover because they are unable to establish a connection to syslog, because it is not healthy immediately after the daemon restarts.

OS: Ubuntu 24.04.2 LTS
Docker version:

Client: Docker Engine - Community
 Version:           28.0.0
 API version:       1.48
 Go version:        go1.23.6
 Git commit:        f9ced58
 Built:             Wed Feb 19 22:11:04 2025
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          28.0.0
  API version:      1.48 (minimum version 1.24)
  Go version:       go1.23.6
  Git commit:       af898ab
  Built:            Wed Feb 19 22:11:04 2025
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Compose plugin: 2.33.0-1~ubuntu.24.04~noble

2

The Docker daemon itself does not deal with dependencies. It just runs containers. Depends on was added to Docker Compose, but compose is a client and it has nothing to do with containers starting after reboot.

Normally you should handle dependencies in the start script of containers and keep it running and waiting until it can connect before you start the main process. The only case it wouldn’t work and which was pointed out by a user in another topic some time ago is when one of your containers is trying to use another container’s network namespace for example, which it cannot before that container starts, so the container cannot be started.

In that case, or if you see the restart policy doesn’t work as expected, if you want to rely on Docker Compose, you can create a Systemd service which runs the docker compose command after the machine booted.

Regarding restart policy not working, I couldn’t reproduce it yet. I haven’t tried hard it though, but I don’t remember it happening only other people reporting it, so I’m not sure what happens.

1 Like