Ubuntu 22.04 unexpected docker socket behaviour after boot

Docker Engine General
1

Hello,

I have 2 identical systems that I setup with Ubuntu 22.04.1 LTS and docker-ce as described in the official documentation (using the official docker packages).

The first system was setup with docker 20.10.21, the second one with 23.0.1. Both systems are working as expected except a difference in behaviour of the second system after boot/reboot.

On the second system I see: the containers that were running before reboot are running after the reboot (I can reach them over the network) and according to systemctl the docker service is running fine. The strange thing is that docker ps gives me an empty response. It is only after a manual restart of the docker.socket service that the containers are restarted and reported when I run ‘docker ps’.

Log of systemctl right after reboot:

user@pp02:~$ sudo systemctl status docker.service
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-02-15 15:02:08 CET; 4min 3s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
   Main PID: 1811 (dockerd)
      Tasks: 85
     Memory: 134.0M
        CPU: 4.443s
     CGroup: /system.slice/docker.service
             ├─1811 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
             ├─2171 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 1883 -container-ip 172.20.0.2 -container-port 1883
             ├─2177 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 1883 -container-ip 172.20.0.2 -container-port 1883
             ├─3862 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 631 -container-ip 172.20.0.20 -container-port 631
             ├─3869 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 631 -container-ip 172.20.0.20 -container-port 631
             ├─3995 /usr/bin/docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 1880 -container-ip 172.20.0.21 -container-port 1880
             └─4001 /usr/bin/docker-proxy -proto tcp -host-ip :: -host-port 1880 -container-ip 172.20.0.21 -container-port 1880

Feb 15 15:02:06 pp02 dockerd[1811]: time="2023-02-15T15:02:06.811724764+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.256351370+01:00" level=info msg="No non-localhost DNS nameservers are left in resolv.conf. Using default external servers: [nameserver 8.8.8.8 nameserver 8.8.4.4]"
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.256382269+01:00" level=info msg="IPv6 enabled; Adding default IPv6 external servers: [nameserver 2001:4860:4860::8888 nameserver 2001:4860:4860::8844]"
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.281264686+01:00" level=warning msg="Security options with `:` as a separator are deprecated and will be completely unsupported in 17.04, use `=` instead."
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.778362015+01:00" level=info msg="Loading containers: done."
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.796451975+01:00" level=info msg="Docker daemon" commit=bc3805a graphdriver=overlay2 version=23.0.1
Feb 15 15:02:07 pp02 dockerd[1811]: time="2023-02-15T15:02:07.796611263+01:00" level=info msg="Daemon has completed initialization"
Feb 15 15:02:08 pp02 dockerd[1811]: time="2023-02-15T15:02:08.343152806+01:00" level=info msg="[core] [Server #7] Server created" module=grpc
Feb 15 15:02:08 pp02 systemd[1]: Started Docker Application Container Engine.
Feb 15 15:02:08 pp02 dockerd[1811]: time="2023-02-15T15:02:08.349127023+01:00" level=info msg="API listen on /run/docker.sock"
user@pp02:~$ sudo systemctl status docker.socket
● docker.socket - Docker Socket for the API
     Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
     Active: active (running) since Wed 2023-02-15 15:02:03 CET; 4min 13s ago
   Triggers: ● docker.service
     Listen: /run/docker.sock (Stream)
      Tasks: 0 (limit: 8947)
     Memory: 0B
        CPU: 1ms
     CGroup: /system.slice/docker.socket

Feb 15 15:02:03 pp02 systemd[1]: Starting Docker Socket for the API...
Feb 15 15:02:03 pp02 systemd[1]: Listening on Docker Socket for the API.

As stated the containers that were running before the reboot are up (I can reach them on their exposed service network ports) but if I attempt to list the containers I get an empty response:

user@pp02:~$ docker ps
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

If I restart the docker.socker service the containers are restarted and listed correctly:

user@pp:~$ date
Wed Feb 15 03:06:51 PM CET 2023
user@pp02:~$ sudo systemctl restart docker.socket
user@pp02:~$ docker ps
CONTAINER ID   IMAGE                         COMMAND                  CREATED        STATUS                            PORTS                                       NAMES
fe014a282f45   eclipse-mosquitto             "/docker-entrypoint.…"   6 days ago     Up 2 seconds                      0.0.0.0:1883->1883/tcp, :::1883->1883/tcp   mosquitto
463b12855319   nodered/node-red:latest       "./entrypoint.sh"        6 days ago     Up 2 seconds (health: starting)   0.0.0.0:1880->1880/tcp, :::1880->1880/tcp   docker-node-red-1
<I have removed the other containers that are running to avoid littering the log, entries are identical to the 2 listed ones above>

I already scanned the forums but could not find people with a similar issue. I tried reverting the docker-ce and docker-ce-cli packages to 20.10.21 (which are the versions on the first computer that is working as expected), but that does not result in the expected behaviour.

The annoying thing is that any docker command I try to execute after reboot seems to think that the containers are not running, although in fact they are. This means that e.g. docker-compose up -d will try to start the mosquitto container again, which will fail because the port is already bound by the mosquitto container that is running ‘behind the scenes’. Also attaching to a container is not possible.

I also already tried without result:

  • reinstalling docker
sudo apt-get remove docker docker-engine docker.io containerd runc docker-ce
sudo apt autoremove
sudo apt install docker-ce docker-ce-cli containerd.io
  • docker system prune
  • sudo systemctl restart docker.service > this does not resolve the issue, only the restart of the docker.socket service helps.

Are there any other pointers somebody can provide where to look further to find and solve the root of this issue?

Your help is much appreciated.

Thanks,
Lieven.

Version information:

user@pp02:~$ docker version
Client: Docker Engine - Community
 Version:           23.0.1
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        a5ee5b1
 Built:             Thu Feb  9 19:47:01 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.1
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       bc3805a
  Built:            Thu Feb  9 19:47:01 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
user@pp02:~$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"
user@pp02:~$ uname -a
Linux piaprog02 5.15.0-60-generic #66-Ubuntu SMP Fri Jan 20 14:29:49 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
2

Adding the logs since boot for docker.service and docker.socket, including the systemctl restart docker.docket command I did to make sure that I can interact with the existing containers as I stated in the previous message.
journalctl-logs.txt (47.6 KB)

3

Based on your observation that downgrading Docker doesn’t help and restarting docker doesn’t help only restarting the docker socket, my guess would be that somehow you have another docker installation which overwrites the socket after reboot and restarting the docker.socket service restores it.

Please, run the following command before and after reboot and share the output:

ls -l /run/docker.sock

docker version

docker context ls

You could also check the output following commands (doesn’T matter when you run them)

dpkg -l | grep docker

snap list | grep docker
1 Like
4

Hello @rimelek

your guess is spot on. Thank you very much for pointing me in the right direction. Apparently the system had two docker installations in parallel: one from the apt store and one from snap. :flushed:

The snap one gets started at boot and the apt store one gets activated after the restart of the docker socket.

Removing the snap one resolves the issue I was seeing.

Thanks again, I learned something new today!

Kind regards,
Lieven.

user@pp02:~$ dpkg -l | grep docker
ii  docker-buildx-plugin                  0.10.2-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
ii  docker-ce                             5:23.0.1-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                         5:23.0.1-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras             5:23.0.1-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                 2.16.0-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  docker-scan-plugin                    0.23.0~ubuntu-jammy                     amd64        Docker scan cli plugin.
hydroko@piaprog02:~$ snap list | grep docker
docker                20.10.17       2343   latest/stable  canonical**  -
1 Like