Healthcheck seems to be broken in Linux

General
1

I have a healthcheck on my mysql service container.

My app container should wait till mysql is marked as “healthy”.

After that the app service will run migrations and that is the problem.

The mysql service is marked as “healthy”, but it isnt and the timmings also look fishy.

Here is my mysql service in the docker-compose.yml:

    mysql:
        image: artifactory.local.xxx.de/wm-container/wm-mysql:8.0
        container_name: gcc-mysql
        command: --local-infile=ON --innodb_file_per_table=ON --innodb_redo_log_capacity=512M --innodb_strict_mode=OFF
        restart: unless-stopped
        ports:
            - '${FORWARD_DB_PORT:-3306}:3306'
        environment:
            TZ: Europe/Berlin
            MYSQL_ROOT_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ROOT_HOST: "%"
            MYSQL_DATABASE: '${DB_DATABASE}'
            MYSQL_USER: '${DB_USERNAME}'
            MYSQL_PASSWORD: '${DB_PASSWORD}'
            MYSQL_ALLOW_EMPTY_PASSWORD: 1
        volumes:
            - 'ggc-mysql:/var/lib/mysql'
        networks:
            - gleif-golden-copy
        healthcheck:
            test: ["CMD", "mysqladmin", "ping", "-p${DB_PASSWORD}"]
            interval: 40s
            timeout: 5s
            retries: 50
            start_period: 50s

As you can see I have a “start_period” off 50s (wich is a bit too much) but my mysql container seems to start in 12s-15s and so my app service gets started and tries to run the migrations, wich will fail, because mysql isnt ready.

image
image2013×176 26.1 KB

This seemed to work the past versions with even lower timmings.

Docker Info:

Client: Docker Engine - Community
 Version:    27.3.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.17.1
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.29.7
    Path:     /usr/libexec/docker/cli-plugins/docker-compose
  scout: Docker Scout (Docker Inc.)
    Version:  v1.14.0
    Path:     /home/alves/.docker/cli-plugins/docker-scout

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 27.3.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc version: v1.1.14-0-g2c9f560
 init version: de40ad0
 Security Options:
  seccomp
   Profile: builtin
 Kernel Version: 5.15.133.1-microsoft-standard-WSL2
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.42GiB
 Name: WXLP609
 ID: 9522c6b8-64c7-41e6-a67a-5f3595f2f581
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No blkio throttle.read_bps_device support
WARNING: No blkio throttle.write_bps_device support
WARNING: No blkio throttle.read_iops_device support
WARNING: No blkio throttle.write_iops_device support

Docker Version:

Client: Docker Engine - Community
 Version:           27.3.1
 API version:       1.47
 Go version:        go1.22.7
 Git commit:        ce12230
 Built:             Fri Sep 20 11:41:03 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.3.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.7
  Git commit:       41ca978
  Built:            Fri Sep 20 11:41:03 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.22
  GitCommit:        7f7fdf5fed64eb6a7caf99b3e12efcf9d60e311c
 runc:
  Version:          1.1.14
  GitCommit:        v1.1.14-0-g2c9f560
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

EDIT:
I am using Docker installed directly under WSL2 (Ubuntu 20.04) without Docker Desktop on Windows.

2

How is your app service set up? Maybe that’s wrong?

Maybe your healthcheck does not work. Not sure if this is related to Docker or simply the command you are trying to run. Should it respond the way the healthcheck expects it?

Having the 50s start period seems like a sneaky trick, why not use 120 sec? :wink:

3

https://dev.mysql.com/doc/refman/8.4/en/mysqladmin.html

  • ping

Check whether the server is available. The return status from mysqladmin is 0 if the server is running, 1 if it is not. This is 0 even in case of an error such as Access denied, because this means that the server is running but refused the connection, which is different from the server not running.

So make sure you use a test command that tells you whether the database is ready for what you want and not just running or test only that the container is running and improve your commands and entrypoints in the containers to start processes when the dependencies are ready.

I would personally just improve the migration script to retry until there is no error. Of course I would check the error code before retrying to avoud rerunning what already ran.

4

That wasn’t a trick at all. That was a way to show that I have set it to 50s but the container still starts at the 12s mark and so my Migrations throw an error.