Docker takes 100% of all CPUs until become unresponsive. High disk writes

OS Version/build

uname -a
Linux host 6.1.0-21-cloud-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.90-1 (2024-05-03) x86_64 GNU/Linux

df -h
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 40G 11G 27G 29% /

iotop
Total DISK READ: 0.00 B/s | Total DISK WRITE: 551.26 K/s
Current DISK READ: 0.00 B/s | Current DISK WRITE: 6.19 M/s
TID PRIO USER DISK READ DISK WRITE> COMMAND
1157386 be/4 root 0.00 B/s 195.38 K/s dockerd -H fd:// --containerd=/run/containerd/containerd.sock
716 be/4 root 0.00 B/s 188.41 K/s dockerd -H fd:// --containerd=/run/containerd/containerd.sock
706 be/4 root 0.00 B/s 167.47 K/s dockerd -H fd:// --containerd=/run/containerd/containerd.sock

App version

sudo docker version
Client: Docker Engine - Community
Version: 24.0.6
API version: 1.43
Go version: go1.20.7
Git commit: ed223bc
Built: Mon Sep 4 12:32:10 2023
OS/Arch: linux/amd64
Context: default

Server: Docker Engine - Community
Engine:
Version: 24.0.6
API version: 1.43 (minimum version 1.12)
Go version: go1.20.7
Git commit: 1a79695
Built: Mon Sep 4 12:32:10 2023
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.24
GitCommit: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc:
Version: 1.1.9
GitCommit: v1.1.9-0-gccaecfc
docker-init:
Version: 0.19.0
GitCommit: de40ad0

docker stats
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
xyz abc 0.12% 402.7MiB / 1.89GiB 20.80% 1.34GB / 176MB 381MB / 22.6MB 21

docker info
Client: Docker Engine - Community
Version: 24.0.6
Context: default
Debug Mode: false
Plugins:
buildx: Docker Buildx (Docker Inc.)
Version: v0.11.2
Path: /usr/libexec/docker/cli-plugins/docker-buildx
compose: Docker Compose (Docker Inc.)
Version: v2.21.0
Path: /usr/libexec/docker/cli-plugins/docker-compose

Server:
Containers: 2
Running: 1
Paused: 0
Stopped: 1
Images: 1
Server Version: 24.0.6
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Using metacopy: false
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: systemd
Cgroup Version: 2
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 61f9fd88f79f081d64d6fa3bb1a0dc71ec870523
runc version: v1.1.9-0-gccaecfc
init version: de40ad0
Security Options:
apparmor
seccomp
Profile: builtin
cgroupns
Kernel Version: 6.1.0-21-cloud-amd64
Operating System: Debian GNU/Linux 12 (bookworm)
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.89GiB
Name: host
ID: xyz
Docker Root Dir: /var/lib/docker
Debug Mode: false
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Steps to reproduce
It happens all the time.

I tried also :

  • Docker 24 running on Ubuntu 24.04 : same issue
  • Scanning for corrupt files :

sudo find /var/lib/docker/containers/ -name *-json.log -exec bash -c ‘jq ‘.’ {} > /dev/null 2>&1 || echo “file corrupt: {}”’ ; I deleted the files but keeps coming back.

  • I tried also to do a system prune, rebuild images but comes back.
  • I tried a reinstall, same thing

How to solve this issue ?