All docker command line commands hang

Docker Engine General
1

Context

Running Ubuntu 16.04.5 LTS. docker version hangs indefinitely with no output, so here’s version info from apt:

$ apt list | grep docker
cockpit-docker/xenial-backports,xenial-backports 178-1~ubuntu16.04.1 all
docker/xenial 1.5-1 amd64
docker-ce/xenial,now 5:18.09.1~3-0~ubuntu-xenial amd64 [installed]
docker-ce-cli/xenial,now 5:18.09.1~3-0~ubuntu-xenial amd64 [installed,automatic]
docker-compose/xenial-updates,xenial-updates 1.8.0-2~16.04.1 all
docker-doc/xenial-updates,xenial-updates 18.06.1-0ubuntu1~16.04.2 all
docker-registry/xenial 2.3.0~ds1-1 amd64
docker.io/xenial-updates 18.06.1-0ubuntu1~16.04.2 amd64

Problem

As stated, the problem is that all docker commands hang indefinitely with no output. Including docker info, docker help, docker version, docker ps, and docker images.

doc@companero-capitan:~ $ docker version
^C
doc@companero-capitan:~ $

I have tried the solution suggested by @korayhk at What to do when all docker commands hang?, as well as that by @samsgates immediately above (including the sudo rm -rf /var/lib/docker and sudo apt remove docker; sudo apt install docker.

I have rebooted.

Any help troubleshooting this would be greatly appreciated.

Diagnostics

service output:

$ sudo service docker status
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
  Drop-In: /etc/systemd/system/docker.service.d
           └─override.conf
   Active: active (running) since Wed 2019-01-30 11:43:20 EST; 15min ago
     Docs: https://docs.docker.com
 Main PID: 4162 (dockerd)
    Tasks: 13
   Memory: 27.8M
      CPU: 704ms
   CGroup: /system.slice/docker.service
           └─4162 /usr/bin/dockerd -D -H tcp://127.0.0.1:2375

Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943463833-05:00" level=debug msg="Registering GET, /distribution/{name:.*}/json"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943539156-05:00" level=debug msg="Registering GET, /networks"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943580359-05:00" level=debug msg="Registering GET, /networks/"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943630631-05:00" level=debug msg="Registering GET, /networks/{id:.+}"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943698691-05:00" level=debug msg="Registering POST, /networks/create"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943749156-05:00" level=debug msg="Registering POST, /networks/{id:.*}/connect"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943825222-05:00" level=debug msg="Registering POST, /networks/{id:.*}/disconnect"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943901449-05:00" level=debug msg="Registering POST, /networks/prune"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.943958877-05:00" level=debug msg="Registering DELETE, /networks/{id:.*}"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.944314533-05:00" level=info msg="API listen on 127.0.0.1:2375"

ps output:

$ ps aux | grep docker
root      4162  0.0  0.8 535264 67324 ?        Ssl  11:43   0:00 /usr/bin/dockerd -D -H tcp://127.0.0.1:2375

dmesg output:

$ dmesg | grep docker
[   10.234758] audit: type=1400 audit(1548863042.525:19): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1412 comm="apparmor_parser"
[   10.522528] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready

pstree output:

$ pstree -aps 4162
systemd,1
  └─dockerd,4162 -D -H tcp://127.0.0.1:2375
      ├─{dockerd},4166
      ├─{dockerd},4167
      ├─{dockerd},4169
      ├─{dockerd},4170
      ├─{dockerd},4171
      ├─{dockerd},4172
      ├─{dockerd},4173
      ├─{dockerd},4175
      ├─{dockerd},4176
      ├─{dockerd},4221
      └─{dockerd},4222

syslog output:

$ cat /var/log/syslog | grep docker | grep -v "level=debug" | tail -60
Jan 30 09:38:36 companero-capitan dockerd[31466]: time="2019-01-30T10:38:36.979229673-05:00" level=info msg="Loading containers: done."
Jan 30 09:38:37 companero-capitan dockerd[31466]: time="2019-01-30T10:38:37.009598900-05:00" level=info msg="Docker daemon" commit=4c52b90 graphdriver(s)=overlay2 version=18.09.1
Jan 30 09:38:37 companero-capitan dockerd[31466]: time="2019-01-30T10:38:37.009715463-05:00" level=info msg="Daemon has completed initialization"
Jan 30 09:38:37 companero-capitan dockerd[31466]: time="2019-01-30T10:38:37.028595912-05:00" level=info msg="API listen on 127.0.0.1:2375"
Jan 30 09:43:13 companero-capitan avahi-daemon[864]: Leaving mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1.
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.487375711-05:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Jan 30 10:44:02 companero-capitan kernel: [   10.234758] audit: type=1400 audit(1548863042.525:19): apparmor="STATUS" operation="profile_load" profile="unconfined" name="docker-default" pid=1412 comm="apparmor_parser"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.534465653-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.534495614-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.534915115-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.534966287-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535032841-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207c8b20, CONNECTING" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535039029-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535056499-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535255178-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535298254-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.535359781-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4208b8140, CONNECTING" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.536182700-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4207c8b20, READY" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.536360330-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4208b8140, READY" module=grpc
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.558494068-05:00" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.643610514-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.643816071-05:00" level=warning msg="Your kernel does not support swap memory limit"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.643861041-05:00" level=warning msg="Your kernel does not support cgroup rt period"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.643873226-05:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.644298947-05:00" level=info msg="Loading containers: start."
Jan 30 10:44:02 companero-capitan avahi-daemon[857]: Joining mDNS multicast group on interface docker0.IPv4 with address 172.17.0.1.
Jan 30 10:44:02 companero-capitan kernel: [   10.522528] IPv6: ADDRCONF(NETDEV_UP): docker0: link is not ready
Jan 30 10:44:02 companero-capitan avahi-daemon[857]: New relevant interface docker0.IPv4 for mDNS.
Jan 30 10:44:02 companero-capitan avahi-daemon[857]: Registering new address record for 172.17.0.1 on docker0.IPv4.
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.894495972-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 30 10:44:02 companero-capitan dockerd[1138]: time="2019-01-30T10:44:02.953082551-05:00" level=info msg="Loading containers: done."
Jan 30 10:44:03 companero-capitan dockerd[1138]: time="2019-01-30T10:44:03.089699047-05:00" level=info msg="Docker daemon" commit=4c52b90 graphdriver(s)=overlay2 version=18.09.1
Jan 30 10:44:03 companero-capitan dockerd[1138]: time="2019-01-30T10:44:03.090799553-05:00" level=info msg="Daemon has completed initialization"
Jan 30 10:44:03 companero-capitan dockerd[1138]: time="2019-01-30T10:44:03.115289507-05:00" level=info msg="API listen on 127.0.0.1:2375"
Jan 30 11:41:54 companero-capitan dockerd[1138]: time="2019-01-30T11:41:54.619365959-05:00" level=info msg="Processing signal 'terminated'"
Jan 30 11:41:54 companero-capitan dockerd[1138]: time="2019-01-30T11:41:54.620298823-05:00" level=error msg="failed to get event" error="rpc error: code = Unavailable desc = transport is closing" module=libcontainerd namespace=moby
Jan 30 11:41:54 companero-capitan dockerd[1138]: time="2019-01-30T11:41:54.620368731-05:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.801744011-05:00" level=warning msg="[!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!]"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804471715-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804486836-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804555069-05:00" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804563429-05:00" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804877099-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804877096-05:00" level=info msg="ccResolverWrapper: sending new addresses to cc: [{unix:///run/containerd/containerd.sock 0  <nil>}]" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804899243-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804900438-05:00" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804948367-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42006e4c0, CONNECTING" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.804947771-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4208d5990, CONNECTING" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.805093567-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc42006e4c0, READY" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.805093558-05:00" level=info msg="pickfirstBalancer: HandleSubConnStateChange: 0xc4208d5990, READY" module=grpc
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.813200896-05:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.813404112-05:00" level=warning msg="Your kernel does not support swap memory limit"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.813438197-05:00" level=warning msg="Your kernel does not support cgroup rt period"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.813447257-05:00" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.813830884-05:00" level=info msg="Loading containers: start."
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.859947682-05:00" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.890672729-05:00" level=info msg="Loading containers: done."
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.926943987-05:00" level=info msg="Docker daemon" commit=4c52b90 graphdriver(s)=overlay2 version=18.09.1
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.927115300-05:00" level=info msg="Daemon has completed initialization"
Jan 30 11:43:20 companero-capitan dockerd[4162]: time="2019-01-30T11:43:20.944314533-05:00" level=info msg="API listen on 127.0.0.1:2375"
2

If I stop the service and then manually start the daemon with sudo dockerd, then docker ... commands work as expected.

3

Solved. I had overridden the docker.service file to start dockerd with arguments -D -H tcp://127.0.0.1:2375. I don’t know what was actually going wrong (maybe that unix:// wasn’t included as a host?) but when I moved the hosts configuration into a /ect/docker/daemon.json file, that solved the problem.

I think the fact that this behavior can arise from the docker client when the host is misconfigured should be treated as a bug.