Docker compose container doesn't stop when healthcheck is supposed to fail

Docker Engine Compose
,
1

I’m running a healthcheck in my docker-compose file, and this healthcheck checks 2 things, 1 is an endpoint, making sure the flask app is up and running, and 2 is pg_isready, checking if the postgres database is running.

This works fine when I run docker compose -up --build, it does the healthcheck, and if pg_isready fails, it doesnt start my service that depends on the healthcheck. But in the case that the healthcheck passes, if later on, the postgres server is shut down, even though pg_isready is failing, the docker containers continue as if nothing is wrong. Shouldn’t my services that depend on the healthcheck fail? I see no restarting or nothing.

Now I’m not even sure if this is a thing, but it would make sense if it is, right? The healthcheck is continuously running all the time, so why does it not have any side effects when it fails later on? Why does the service that depends on the healthcheck, not fail when healthcheck is failing later on?

Here’s my docker-compose file:

services:
  postgres:
    build:
      context: .
      dockerfile: Dockerfile.app1
    ports:
      - 8005:8005
    volumes:
      - ./app:/app
      - D:/Myprojects/PostgresBackups:/app/backup
      - C:/users/bob/onedrive/backups:/app/cloud
    env_file:
      - .env
    networks:
      - my-network
    environment:
      - BACKUP_DIRECTORY=/app/backup
      - PYTHONUNBUFFERED=1
    restart: always
    healthcheck:
      test:
        [
          "CMD-SHELL",
          "curl -f http://postgres:8005/health && pg_isready -U postgres -p 5432 -h host.docker.internal || exit 1",
        ]
      interval: 5s
      timeout: 5s
      retries: 20

  scheduler:
    build:
      context: .
      dockerfile: Dockerfile.app2
    ports:
      - "8006:8006"
    volumes:
      - ./app:/app
    env_file:
      - .env
    networks:
      - my-network
    environment:
      - PYTHONUNBUFFERED=1
    restart: always
    depends_on:
      postgres:
        condition: service_healthy

networks:
  my-network:
2

The healthcheck itself just marks a container as health or unhealthy, but it is not responsible to exit or restart of a container marked as unhealthy.

The answers to this are unpleasant:

  • you can handle this inside your application code itself, that it a) performs a self health check and b) exits the container if the healthcheck fails a certain number of times in a row.
  • You could use https://hub.docker.com/r/willfarrell/autoheal/.
  • You could implement something yourself that listens on the docker event stream, and handles unhealthy containers (like autoheal does).

Kubernetes has a Liveness Probe for Pods, which does exactly what you are looking for. Though, there is nothing comparable in docker itself.

1 Like
3

From the doc:

You can control the order of service startup and shutdown with the depends_on attribute. Compose always starts and stops containers in dependency order,

That sounds to me it only works when running compose to start and stop the full stack, not when a single container fails.

Personally I am not a big fan of depends_on in a world of running independent component like microservices. Each component should handle failure of depend services itself, not tear down the full stack.