Docker hangs on start and does not respond. Full output, stacktrace, config and environment inside

Hello, I’m having an issue with my docker install. When I start the daemon, either through systemd or manually, it will hang and not respond to any commands. I have exhausted every thread I could find and I’m still unable to fix this issue.

I have narrowed it down to the storage driver. When I start docker to use overlay it starts just fine (except for not seeing my containers), but overlay2 does not.

It used to work on this machine however after a reboot it quit working. The reboot breaking it and the problem being related to storage driver makes me think this is a kernel issue. One thread I found recommended installing linux-modules-extra-... and linux-image-extra-virtual However that made no difference. I have also let it sit overnight to see if it would finally load and it does not, the only way to get it to stop is kill -9 .

I’m using docker-ce from bionic/stable amd64

I have not dist-upgraded or made any major changes. Disk not full 10%. There is plenty of ram and cpu resources available, this is the only purpose of this machine. It is also important to me that I am able to recover the volume data I have so I don’t want to clear /var/lib/docker

Any help would be much appreciated, thank you in advance! :slight_smile:


OS: Ubuntu Server 18.04.2
Kernel: 4.15.0-55-generic
Filesystem: ext4 on lvm
Docker: 18.09.1
Containerd: 1.2.2

Also tried:
Docker: 19.03.1
Conatinerd: 1.2.6

Filesystem                  Type      Size  Used Avail Use% Mounted on
udev                        devtmpfs  3.9G     0  3.9G   0% /dev
tmpfs                       tmpfs     798M 1020K  797M   1% /run
/dev/mapper/host--vg-root   ext4      125G   12G  107G  10% /
tmpfs                       tmpfs     3.9G     0  3.9G   0% /dev/shm
tmpfs                       tmpfs     5.0M     0  5.0M   0% /run/lock
tmpfs                       tmpfs     3.9G     0  3.9G   0% /sys/fs/cgroup
/dev/loop0                  squashfs   89M   89M     0 100% /snap/core/7270
/dev/loop2                  squashfs   11M   11M     0 100% /snap/kubectl/1014
/dev/loop1                  squashfs   89M   89M     0 100% /snap/core/7169
/dev/loop3                  squashfs   11M   11M     0 100% /snap/kubectl/1034
tmpfs                       tmpfs     798M     0  798M   0% /run/user/1001

My /etc/docker/daemon.json is usually empty but I have this in it for testing:

  "storage-driver": "overlay2"

Here is what it looks like starting dockerd with debug mode:

$ sudo /usr/bin/dockerd --containerd=/run/containerd/containerd.sock -D -l "debug"
INFO[2019-07-26T22:11:23.838982560-07:00] Starting up
DEBU[2019-07-26T22:11:23.839532697-07:00] Listener created for HTTP on unix (/var/run/docker.sock)
DEBU[2019-07-26T22:11:23.840298306-07:00] Golang's threads limit set to 56970
INFO[2019-07-26T22:11:23.840878724-07:00] parsed scheme: "unix"                         module=grpc
INFO[2019-07-26T22:11:23.840914311-07:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-26T22:11:23.840939245-07:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] }  module=grpc
INFO[2019-07-26T22:11:23.840963140-07:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-26T22:11:23.841201833-07:00] pickfirstBalancer: HandleSubConnStateChange: 0xc00014c910, CONNECTING  module=grpc
INFO[2019-07-26T22:11:23.841231668-07:00] blockingPicker: the picked transport is not ready, loop back to repick  module=grpc
INFO[2019-07-26T22:11:23.841878383-07:00] pickfirstBalancer: HandleSubConnStateChange: 0xc00014c910, READY  module=grpc
INFO[2019-07-26T22:11:23.842936134-07:00] parsd scheme: "unix"                         module=grpc
INFO[2019-07-26T22:11:23.842969892-07:00] scheme "unix" not registered, fallback to default scheme  module=grpc
INFO[2019-07-26T22:11:23.842988017-07:00] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0  <nil>}] }  module=grpc
INFO[2019-07-26T22:11:23.843004117-07:00] ClientConn switching balancer to "pick_first"  module=grpc
INFO[2019-07-26T22:11:23.843087387-07:00] pickfirstBalancer: HandleSubConnStateChange: 0xc0008b2200, CONNECTING  module=grpc
INFO[2019-07-26T22:11:23.843131593-07:00] blockingPicker: the picked transport is not ready, loop back to repick  module=grpc
INFO[2019-07-26T22:11:23.843655723-07:00] pickfirstBalancer: HandleSubConnStateChange: 0xc0008b2200, READY  module=grpc
DEBU[2019-07-26T22:11:23.844395838-07:00] Using default logging driver json-file
DEBU[2019-07-26T22:11:23.844437011-07:00] [graphdriver] trying provided driver: overlay2
DEBU[2019-07-26T22:11:23.844552115-07:00] processing event stream                       module=libcontainerd namespace=plugins.moby
DEBU[2019-07-26T22:11:23.847493678-07:00] backingFs=extfs, projectQuotaSupported=false, indexOff="index=off,"  storage-driver=overlay2
DEBU[2019-07-26T22:11:23.847586895-07:00] Initialized graph driver overlay2
WARN[2019-07-26T22:11:23.862607744-07:00] Your kernel does not support cgroup rt period
WARN[2019-07-26T22:11:23.862646492-07:00] Your kernel does not support cgroup rt runtime
DEBU[2019-07-26T22:11:23.862849606-07:00] Max Concurrent Downloads: 3
DEBU[2019-07-26T22:11:23.862873116-07:00] Max Concurrent Uploads: 5
INFO[2019-07-26T22:11:23.862904180-07:00] Loading containers: start.
DEBU[2019-07-26T22:11:23.863056712-07:00] processing event stream                       module=libcontainerd namespace=moby
DEBU[2019-07-26T22:11:23.864709218-07:00] Loaded container e98fbec425363fcff163384a3bbb1587c423b2076cf9313b06d1e0e3a91343a5, isRunning: false
DEBU[2019-07-26T22:11:23.864732607-07:00] Loaded container 296303298f919606588c762dff121c667d98eae035970e668e492e7063553070, isRunning: false
DEBU[2019-07-26T22:11:23.864734745-07:00] Loaded container 7f94c91d80c605b51e0eeb0a080359008074a7bfd77d690a5c1aa181d2b738b2, isRunning: false
DEBU[2019-07-26T22:11:23.864796772-07:00] Loaded container 92e4f3855a60df74a378425fc4e920d0473d9617f4e2f74c2303b720d1eb6610, isRunning: false
DEBU[2019-07-26T22:11:23.864932555-07:00] Loaded container 1ac354cfbc1c76165138be8aff3e7d82f48392d53ce25718a9ca498ba4e260b9, isRunning: false
DEBU[2019-07-26T22:11:23.864945033-07:00] Loaded container c5e09e4f9de2337c244221b0fc0e40bed364962eb4cf40efb66ccaa8eaf3f086, isRunning: false
DEBU[2019-07-26T22:11:23.864996059-07:00] Loaded container cebe07694e7857ac4ae4f6bacfde2831ed538ee2dd9ee7248381b315626e7019, isRunning: false
^CINFO[2019-07-26T22:16:14.389013997-07:00] Processing signal 'interrupt'
^CINFO[2019-07-26T22:16:14.757979442-07:00] Processing signal 'interrupt'
^CINFO[2019-07-26T22:16:14.929377508-07:00] Processing signal 'interrupt'
^CINFO[2019-07-26T22:16:15.323824421-07:00] Processing signal 'interrupt'

Here is a stack trace when it is hung:

Facing a similar problem. Anyone knows?