Docker fails to stop, restart or kill all containers

Docker Engine General
1

Containers can be created as normal with “docker compose up -d” but when it comes time to manage any container ("compose down, docker stop, docker kill) all commands hang and fail when trying to stop any container.

Since this problem has occurred the only way to stop containers is “sudo systemctl stop docker.socket”. I’ve ran “sudo var/lib/docker/nuke-graph-directory.sh” which cleared all of /var/lib/docker removing all of the unkillable containers. But the inability to properly manage them remains, with new containers not able to be managed.

Any ideas on what is causing this behavior?

2

I am afraid you have to share way more information in order for someone to get an idea of an idea of what might cause the behavior.

We usually need the following information to understand the issue:

  1. What platform are you using? Windows, Linux or macOS? Which version of the operating systems? In case of Linux, which distribution?

  2. How did you install Docker? Sharing the platform almost answers it, but only almost. Direct links to the followed guide can be useful.

  3. On debian based Linux, the following commands can give us some idea and recognize incorrectly installed Docker:

    docker info
docker version

    Review the output before sharing and remove confidential data if any appears (public IP for example)

    dpkg -l 'docker*' | grep '^ii'
snap list docker

    When you share the outputs, always format your posts according to the following guide: How to format your forum posts

3

OS: Debian 12: 6.1.0-28-amd64 Debian 6.1.119-1 (2024-11-22) x86_64
installed via APT

**~$ docker info**
Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 2
  Running: 0
  Paused: 0
  Stopped: 2
 Images: 2
 Server Version: 27.4.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-28-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.4GiB
 Name: ziodberg
 ID: 4aa1b9a6-4dbd-4060-94f7-0d2517a8c5b9
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: s13junky
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

**~$ docker version**
Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:56 2024
 OS/Arch:           linux/amd64
 Context:           default
Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:56 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

**~$ dpkg -l 'docker*' | grep '^ii'**
ii  docker                    1.5-2                         all          transitional package
ii  docker-buildx-plugin      0.19.3-1~debian.12~bookworm   amd64        Docker Buildx cli plugin.
ii  docker-ce                 5:27.4.1-1~debian.12~bookworm amd64        Docker: the open-source application container engine
ii  docker-ce-cli             5:27.4.1-1~debian.12~bookworm amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:27.4.1-1~debian.12~bookworm amd64        Rootless support for Docker.
ii  docker-compose-plugin     2.32.1-1~debian.12~bookworm   amd64        Docker Compose (V2) plugin for the Docker CLI.
4

Still struggling with this even after reinstalling the OS and docker.
setup:

  • installed debian 12.8.0 via DVD iso
  • installed docker via Docker Docs Apt method
  • copied ~/compose folder to new setup (docker-compose.ymls, .envs, and configs)
  • copied /opt/docker/, my container volumes
  • pulled all images before attempting compose up

DOCKER PACKAGES

ii  docker-buildx-plugin      0.19.3-1~debian.12~bookworm   amd64        Docker Buildx cli plugin.
ii  docker-ce                 5:27.4.1-1~debian.12~bookworm amd64        Docker: the open-source application container engine
ii  docker-ce-cli             5:27.4.1-1~debian.12~bookworm amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras 5:27.4.1-1~debian.12~bookworm amd64        Rootless support for Docker.
ii  docker-compose-plugin     2.32.1-1~debian.12~bookworm   amd64        Docker Compose (V2) plugin for the Docker CLI.

DOCKER INFO

Client: Docker Engine - Community
 Version:    27.4.1
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.19.3
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.32.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 11
  Running: 1
  Paused: 0
  Stopped: 10
 Images: 33
 Server Version: 27.4.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 88bf19b2105c8b17560993bee28a01ddc2f97182
 runc version: v1.2.2-0-g7cb3632
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.1.0-28-amd64
 Operating System: Debian GNU/Linux 12 (bookworm)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.4GiB
 Name: Zoidberg
 ID: a2311731-4f6a-4aab-b2ba-5fcca69de7c5
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

DOCKER VERSION

Client: Docker Engine - Community
 Version:           27.4.1
 API version:       1.47
 Go version:        go1.22.10
 Git commit:        b9d17ea
 Built:             Tue Dec 17 15:45:56 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          27.4.1
  API version:      1.47 (minimum version 1.24)
  Go version:       go1.22.10
  Git commit:       c710b88
  Built:            Tue Dec 17 15:45:56 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.7.24
  GitCommit:        88bf19b2105c8b17560993bee28a01ddc2f97182
 runc:
  Version:          1.2.2
  GitCommit:        v1.2.2-0-g7cb3632
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

TEST
bring up maintenance stack
docker-compose.yml

x-logging: &loki-logging
  driver: loki
  options:
    loki-url: "http://127.0.0.1:3100/loki/api/v1/push"
    loki-batch-size: "400"
    keep-file: "true"

services:
  ########################### PORTAINER    ########################
  portainer:
    image: portainer/portainer-ce
    container_name: portainer
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    logging: *loki-logging 
    #command: -H tcp://dockersocket:2375
    networks:
      - proxy
      - maint-socket
    #dns:  
    #  - 192.168.137.1
    ports:
      - 8443:9443
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - $APPDATA/portainer:/data
    labels:
      wud.tag.include: '^\d+\.\d+\.\d+-alpine$$'
      wud.link.template: 'https://github.com/dani-garcia/vaultwarden/releases/tag/$${major}.$${minor}.$${patch}'
      # TRAEFIK
      traefik.enable: true
      traefik.http.routers.portainer.entrypoints: https
      traefik.http.services.portainer.loadbalancer.server.port: 9000
      #local
      traefik.http.routers.portainer.rule: Host(`portainer.$DOMAIN`)
      traefik.http.routers.portainer.middlewares: local-ipallowlist, https-redirect@file,auth #,authentik #redirect and local only
      traefik.http.routers.portainer.tls: true
      #remote
      # traefik.http.routers.portainer-remote.rule: Host(`portainer.bachelor-chow.com`)
      # traefik.http.routers.portainer-remote.middlewares: authentik,portainer-https-redirect
      # traefik.http.routers.portainer-remote.entrypoints: https
      # traefik.http.routers.portainer-remote.tls: true
      # traefik.http.routers.portainer-remote.service: portainer
      # HOMEPAGE
      homepage.group: Maintenance
      homepage.name: Portainer
      homepage.icon: portainer.png
      homepage.instance.external.href: https://portainer.$DOMAIN
      homepage.instance.maint.href: https://$SERVERIP:8443
      homepage.description: Pontainer Managmnet
      homepage.widget.type: portainer
      homepage.widget.url: https://$SERVERIP:8443
      homepage.widget.env: 2
      homepage.widget.key: ptr_jnesqHM7H2oMLaBJWIGn0xnrGDiK3d02P5364rW6+hA=
  ########################## portainer socket
  dockersocket:
    container_name: dockersocket-maint
    image: tecnativa/docker-socket-proxy
    logging: *loki-logging 
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    networks:
      - maint-socket
    environment:
      CONTAINERS: 1
      POST: 1
    privileged: true
    restart: unless-stopped
############################# AUTOHEAL     ########################
  autoheal:
    image: willfarrell/autoheal:latest
    container_name: autoheal
    restart: always
    logging: *loki-logging 
    tty: true
    environment:
      - AUTOHEAL_CONTAINER_LABEL=all #all running containers monitored
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
    #putting on local network broke it
############################# KUMA         ########################       
  kuma:
    image: louislam/uptime-kuma:latest
    container_name: kuma
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    logging: *loki-logging 
    networks:
      - proxy
      - maint-socket
    ports:
      - 3001:3001
    environment:
      - TZ=$TZ
      #- PUID=$KUMA
      #- PGID=$KUMA
    volumes:  
      - $APPDATA/kuma:/app/data
      - /var/run/docker.sock:/var/run/docker.sock #easy monitoy all docker containers
    labels:
      #Settings
      traefik.enable: true
      traefik.http.services.kuma.loadbalancer.server.port: 3001
      #local
      traefik.http.routers.kuma.rule: Host(`kuma.$DOMAIN`)
      traefik.http.routers.kuma.middlewares: https-redirect@file,local-ipallowlist,auth #redirect and local only
      traefik.http.routers.kuma.entrypoints: https
      traefik.http.routers.kuma.tls: true
      #remote
      # traefik.http.routers.kuma-remote.rule: Host(`kuma.bachelor-chow.com`)
      # traefik.http.routers.kuma-remote.middlewares: authentik,kuma-https-redirect
      # traefik.http.routers.kuma-remote.entrypoints: https
      # traefik.http.routers.kuma-remote.tls: true
      # Homepage
      homepage.group: Maintenance
      homepage.name: Uptime Kuma
      homepage.icon: uptime-kuma.png
      homepage.weight: 2
      homepage.instance.maint.href: http://$SERVERIP:3001
      homepage.instance.external.href: http://kuma.$DOMAIN
      homepage.description: Uptime Monitoring
      homepage.widget.type: uptimekuma
      homepage.widget.url: http://$SERVERIP:3001
      homepage.widget.key: uk1_Fa8GYdnJ5qAA_rIQRRKvuvU_N1M6JVM9b4ZojFsr
      homepage.widget.slug: slug 
#################### SCRUTINY ############################
  scrutiny:
    container_name: scrutiny
    image: ghcr.io/analogj/scrutiny:master-omnibus
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    logging: *loki-logging 
    cap_add:
      - SYS_RAWIO
      - SYS_ADMIN
    ports:
      - 8081:8080 # webapp
      #- 8086:8086 # influxDB admin
    environment:
      - TZ=$TZ
      - PUID=926 # SCRUTINY
      - GUID=926
    networks:
      - proxy
    volumes:
      - /run/udev:/run/udev:ro
      - $APPDATA/scrutiny/config:/opt/scrutiny/config
      - $APPDATA/scrutiny/influxdb:/opt/scrutiny/influxdb
    devices:
      - "/dev/nvme0"
      - "/dev/sdb"
      - "/dev/sda"
      - "/dev/sdc"
    labels:
      #Settings
      traefik.enable: true
      traefik.http.services.scrutiny.loadbalancer.server.port: 8080
      #local
      traefik.http.routers.scrutiny.rule: Host(`scrutiny.$DOMAIN`)
      traefik.http.routers.scrutiny.middlewares: https-redirect@file,local-ipallowlist,auth #redirect and local only
      traefik.http.routers.scrutiny.entrypoints: https
      traefik.http.routers.scrutiny.tls: true
      #remote
      # traefik.http.routers.scrutiny-remote.rule: Host(`scrutiny.bachelor-chow.com`)
      # traefik.http.routers.scrutiny-remote.middlewares: authentik,scrutiny-https-redirect
      # traefik.http.routers.scrutiny-remote.entrypoints: https
      # traefik.http.routers.scrutiny-remote.tls: true
      homepage.group: Maintenance
      homepage.name: scrutiny
      homepage.icon: scrutiny.png
      homepage.instance.internal.href: https://scrutiny.$DOMAIN/
      homepage.instance.maint.href: https://$SERVERIP:8081
      homepage.description: Drive Health
      homepage.widget.type: scrutiny
      homepage.widget.url: http://$SERVERIP:8081
##################### HOMEPAGE MAINT ######################      
  homepage:
    image: ghcr.io/gethomepage/homepage:latest
    container_name: homepage-maint
    restart: unless-stopped
    security_opt:
      - no-new-privileges=true
    logging: *loki-logging 
    ports:
      - 3006:3000
    environment:
      - TZ=$TZ
      - PUID=925 #homepage
      - PGID=897 #services so i can edit configs easy
      - LOG_LEVEL=debug      
    networks:
      - maint-socket
      - proxy
    volumes:
      - ./homepage/maint/:/app/config # Make sure your config directory exists
      - ./homepage/maint/images/:/app/public/images #loads self hosted images
      - /data:/data:ro # to check space
      - /download:/download:ro
      - /mnt/intake:/imported:ro
############################## QDIRSTAT ######################################      
  # qdirstat:
  #   image: lscr.io/linuxserver/qdirstat:latest
  #   container_name: qdirstat
  #   environment:
  #     - PUID=1000
  #     - PGID=1000
  #     - TZ=$TZ
  #   volumes:
  #     - $APPDATA/qdirstat/config:/config
  #     - /data:/data
  #   networks:
  #     - proxy
  #   ports:
  #     - 3004:3000
  #     - 3005:3001
  #   restart: unless-stopped
  #   # labels:
  #   #   #Settings
  #   #   traefik.enable: true
  #   #   traefik.http.routers.qdirstat.entrypoints: http
  #   #   traefik.http.middlewares.qdirstat-https-redirect.redirectscheme.scheme: https
  #   #   traefik.http.services.qdirstat.loadbalancer.server.port: 3000
  #   #   #local
  #   #   traefik.http.routers.qdirstat.rule: Host(`qdirstat.local.bachelor-chow.com`)
  #   #   traefik.http.routers.qdirstat.middlewares: qdirstat-https-redirect,local-ipallowlist #redirect and local only
  #   #   traefik.http.routers.qdirstat-secure.rule: Host(`qdirstat.local.bachelor-chow.com`)
  #   #   traefik.http.routers.qdirstat-secure.middlewares: local-ipallowlist #In house only
  #   #   traefik.http.routers.qdirstat-secure.entrypoints: https
  #   #   traefik.http.routers.qdirstat-secure.tls: true
  #   #   traefik.http.routers.qdirstat-secure.service: qdirstat
  #   #   #remote
  #   #   traefik.http.routers.qdirstat-remote.rule: Host(`qdirstat.bachelor-chow.com`)
  #   #   traefik.http.routers.qdirstat-remote.middlewares: authentik,qdirstat-https-redirect
  #   #   traefik.http.routers.qdirstat-remote.entrypoints: https
  #   #   traefik.http.routers.qdirstat-remote.tls: true
  #   #   traefik.http.routers.qdirstat-remote.service: qdirstat
############################# Whats up docker ########################
  whatsupdocker:
    image: getwud/wud
    container_name: whatsupdocker
    logging: *loki-logging 
    # healthcheck:
    #   test: wget --no-verbose --tries=1 --no-check-certificate --spider http://localhost:3000
    #   interval: 10s
    #   timeout: 10s
    #   retries: 3
    #   start_period: 10s 
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - $APPDATA/wud/store:/store
    ports:
      - 3007:3000
    networks:
      #- maint-socket
      - proxy
    environment:
      # WUD_WATCHER_LOCAL_HOST: http://dockersocket-maint:2375
      # WUD_WATCHER_LOCAL_PORT: 2375
      WUD_LOG_FORMAT: json # FOR LOGI LOGS
      WUD_REGISTRY_HUB_PUBLIC_LOGIN: s13junky
      WUD_REGISTRY_HUB_PUBLIC_TOKEN: dckr_pat_cNKEoM6qd-fCHNT23v31uG-aL3U
      WUD_WATCHER_LOCAL_CRON: "0 12 * * FRI"
    labels:
      wud.tag.include: '^\d+\.\d+\.\d+$$'
      wud.link.template: 'https://github.com/getwud/wud/releases/tag/$${major}.$${minor}.$${patch}'
      #Settings
      traefik.enable: true
      traefik.http.services.wud.loadbalancer.server.port: 3007
      #local
      traefik.http.routers.wud.rule: Host(`wud.$DOMAIN`)
      traefik.http.routers.wud.middlewares: https-redirect@file,auth #,local-ipallowlist #redirect and local only
      traefik.http.routers.wud.entrypoints: https
      traefik.http.routers.wud.tls: true
      #traefik.http.routers.wud.service: wud
      #remote
      # traefik.http.routers.wud-remote.rule: Host(`wud.$DOMAIN`)
      # traefik.http.routers.wud-remote.middlewares: authentik,https-redirect@file
      # traefik.http.routers.wud-remote.entrypoints: http
      # traefik.http.routers.wud-remote.tls: true
      #traefik.http.routers.wud-remote.service: wud
      # Homepage
      homepage.group: Maintenance
      homepage.name: What's Up Docker
      homepage.icon: whats-up-docker.png
      homepage.weight: 3
      homepage.instance.external.href: https://wud.$DOMAIN
      homepage.instance.maint.href: http://$SERVERIP:3007
      homepage.description: Container Updates
      homepage.widget.type: whatsupdocker
      homepage.widget.url: http://$SERVERIP:3007
#######################################################################################
######                                                                           ######
######                        NETWORKS                                           ######
######                                                                           ######
#######################################################################################
networks:
  proxy:  #OUTSIDE SHARE ABLE
    external: true
  maint-socket:
    external: false  
  #vault: #locks down all containers with root priv not needing network access
  #  external: true

Attempted to bring up, then bring down with compose, then checked status.

docker compose up -d
docker compose down
docker ps

Bringing the containers up takes a little longer then expected ~60sec, but going down i terminated the process after 3 mins. This much much slower then I’m used to as everything but media is on NVME. docker ps shows all containers in the stack are still up.

I tried to bring up the plex stack (plex and tautulli) but containers took over 3 mins to start. So I terminated bringing them up. Ran ````docker psagain, none of plex stack was up but some of the maintenance stack stopped. I randocker ps``` a while latter and all of the maintenance stack was down and the complete plex stack was up. It seems like like docker is lagging or hanging up some where.

Once the the maintenance stack was down i retested docker compose up with btop running in another ssh terminal

After the containers were up, showing log output I terminated compose. Normally containers stop in seconds, it took 390 seconds to just to stop the docker socket proxy the rest took more! In btop you could see /usr/libexec/docker/cli-plugins/docker-compose compose up 1 core to 100% the whole time with the rest of the 11 cores idle. What in the world could be causing compose to hang this bad?

5

There were two cases I experienced slow Docker commands

  • Antivirus installed on the Linux host and checking the docker data root where containers and images are stored. In this case the antivirus has to be configured to ignore the docker data root
  • When there was a problem on a physical host, possibly with software RAID configuration and basic IO operations slowed down. Not just Docker but the whole host. It was never proven, just suspected

If everything is fast, but Docker is slow, I would say the issue is an antivirus or any software that blocks Docker for minutes. It could be a network issue if the images are pulled from a registry and still not existing on the Docker host. If the host is slow too, then maybe it is not Docker where you should look for the issue.