Apt upgrade on Raspberry Pi4 broke docker when it upgraded to 23.0.0

pluim003 · February 2, 2023, 4:52pm

On my 64bit Raspberry Pi4 I did a sudo apt update and sudo apt upgrade. After that the Docker-service wouldn’t start anymore. Docker version showed:

root@raspberrypi4:/etc/apt/sources.list.d# docker version
Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:44:38 2023
 OS/Arch:           linux/arm64
 Context:           default
Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

On my still functioning Raspberry Pi2 I have:

pi@raspberrypi2:~ $ docker version
Client:
 Version:           20.10.5+dfsg1
 API version:       1.41
 Go version:        go1.15.15
 Git commit:        55c4c88
 Built:             Mon May 30 18:34:49 2022
 OS/Arch:           linux/arm
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.5+dfsg1
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.15
  Git commit:       363e9a8
  Built:            Mon May 30 18:34:49 2022
  OS/Arch:          linux/arm
  Experimental:     false
 containerd:
  Version:          1.4.13~ds1
  GitCommit:        1.4.13~ds1-1~deb11u3
 runc:
  Version:          1.0.0~rc93+ds1
  GitCommit:        1.0.0~rc93+ds1-5+deb11u2
 docker-init:
  Version:          0.19.0
  GitCommit:

More logging:

root@raspberrypi4:/etc/apt/sources.list.d# journalctl -eu docker
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/libnetwork/drvregistry/drvregistry.go:72
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/libnetwork.New({0x4000623730, 0x8, 0xe})
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/libnetwork/controller.go:217 +0x514
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/daemon.(*Daemon).initNetworkController(0x40004a98c0, 0x4000cea9c0)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/daemon/daemon_unix.go:851 +0x3c
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/daemon.(*Daemon).restore(0x40004a98c0)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/daemon/daemon.go:478 +0x414
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/daemon.NewDaemon({0x556f9d5018?, 0x400004e4c0}, 0x4000562000, 0x40007>
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/daemon/daemon.go:1085 +0x23a4
feb 02 17:20:36 raspberrypi4 dockerd[9638]: main.(*DaemonCli).start(0x4000a586c0, 0x4000550ee0)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:200 +0x7e8
feb 02 17:20:36 raspberrypi4 dockerd[9638]: main.runDaemon(...)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
feb 02 17:20:36 raspberrypi4 dockerd[9638]: main.newDaemonCommand.func1(0x4000566000?, {0x4000a58660?, 0x3?, 0x3?})
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:38 +0x68
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0x4000566000, {0x400>
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:916 +0x5c8
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0x4000566000)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:1044 +0x35c
feb 02 17:20:36 raspberrypi4 dockerd[9638]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(...)
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:968
feb 02 17:20:36 raspberrypi4 dockerd[9638]: main.main()
feb 02 17:20:36 raspberrypi4 dockerd[9638]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:102 +0x170

● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2023-02-02 17:12:47 CET; 2min 55s ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 7753 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)
   Main PID: 7753 (code=exited, status=2)
        CPU: 2.945s

Then I decided to downgrade Docker and found a 6 yo thread to do so:

root@raspberrypi4:/etc/apt/sources.list.d#  apt-cache policy docker-ce
docker-ce:
  Geïnstalleerd: 5:23.0.0-1~debian.11~bullseye
  Kandidaat:     5:23.0.0-1~debian.11~bullseye
  Versietabel:
 *** 5:23.0.0-1~debian.11~bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
        100 /var/lib/dpkg/status
     5:20.10.23~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.22~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.21~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.20~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.19~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.18~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.17~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.16~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.15~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.14~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.13~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.12~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.11~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.10~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.9~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.8~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.7~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages
     5:20.10.6~3-0~debian-bullseye 500
        500 https://download.docker.com/linux/debian bullseye/stable arm64 Packages

root@raspberrypi4:/etc/apt/sources.list.d# apt install --allow-downgrades -y docker-ce=5:20.10.22~3-0~debian-bullseye
Pakketlijsten worden ingelezen... Klaar
Boom van vereisten wordt opgebouwd... Klaar
De statusinformatie wordt gelezen... Klaar
Voorgestelde pakketten:
  cgroupfs-mount | cgroup-lite
Aanbevolen pakketten:
  apparmor
De volgende pakketten zullen GEDEGRADEERD worden:
  docker-ce
0 opgewaardeerd, 0 nieuw geïnstalleerd, 1 gedegradeerd, 0 te verwijderen en 0 niet opgewaardeerd.
1 niet volledig geïnstalleerd of verwijderd.
Er moeten 12,7 MB aan archieven opgehaald worden.
Na deze bewerking zal er 6.762 kB schijfruimte vrijkomen.
Ophalen:1 https://download.docker.com/linux/debian bullseye/stable arm64 docker-ce arm64 5:20.10.22~3-0~debian-bullseye [12,7 MB]
12,7 MB opgehaald in 2s (6.075 kB/s)
dpkg: waarschuwing: docker-ce wordt gedegradeerd van 5:23.0.0-1~debian.11~bullseye naar 5:20.10.22~3-0~debian-bullseye
(Database wordt ingelezen ... 94220 bestanden en mappen momenteel geïnstalleerd.)
Uitpakken van .../docker-ce_5%3a20.10.22~3-0~debian-bullseye_arm64.deb wordt voorbereid...
Bezig met uitpakken van docker-ce (5:20.10.22~3-0~debian-bullseye) over (5:23.0.0-1~debian.11~bullseye) ...
Instellen van docker-ce (5:20.10.22~3-0~debian-bullseye) ...
root@raspberrypi4:/etc/apt/sources.list.d# ps -ef | grep docker
root       12419       1 11 17:33 ?        00:00:06 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock
...

and luckily my containers started nicely.

root@raspberrypi4:/etc/apt/sources.list.d# docker version
Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.41 (downgraded from 1.42)
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:44:38 2023
 OS/Arch:           linux/arm64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          20.10.22
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.18.9
  Git commit:       42c8b31
  Built:            Thu Dec 15 22:25:43 2022
  OS/Arch:          linux/arm64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

So in case someone encounters the same problem(s). This is a how-to. Took me about 20-30 minutes to solve and get my containers back up and running.

meyay · February 2, 2023, 7:22pm

Thank you for sharing your experience and your how-to. I moved it to the Tips & HowTos category.

Just updated it on Ubuntu 20.04 on a x86_64 system: it took a couple of seconds after reboot that the docker engine was reachable. Apart from that, it appears like it worked like a charm.

pluim003 · February 2, 2023, 7:36pm

I also did a reboot but that didn’t help either. Only way, for now, to got it working again was to downgrade. Maybe there are other solutions but didn’t had the time to dig into this.

meyay · February 2, 2023, 7:45pm

Sure, just because it works on x86_64 architecture, doesn’t mean the releases for each and every architecture provide the same experience.

So take your time with migrating to a recent version, unless of course there is a security fix that doesn’t exist for the 20.04.x versions or provides a feature you need for a particular configuration. I just went with the update to see whether it works for x86_64

curiousitpeter · February 2, 2023, 10:49pm

Unfortunately the same update broke my Docker Engine on Ubuntu 20.04 x86_64, so it does indeed affect x86_64

meyay · February 2, 2023, 10:57pm

So apparently it’s not even the same experience for the same architecture

This is docker version for me:

Client: Docker Engine - Community
 Version:           23.0.0
 API version:       1.42
 Go version:        go1.19.5
 Git commit:        e92dd87
 Built:             Wed Feb  1 17:49:08 2023
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          23.0.0
  API version:      1.42 (minimum version 1.12)
  Go version:       go1.19.5
  Git commit:       d7573ab
  Built:            Wed Feb  1 17:49:08 2023
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.16
  GitCommit:        31aa4358a36870b21a992d3ad2bef29e1d693bec
 runc:
  Version:          1.1.4
  GitCommit:        v1.1.4-0-g5fd4c4d
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

neersighted · February 6, 2023, 7:06am

From the partial stack trace, this looks like it could be panic: interface {} is nil in libnetwork/drivers/ipvlan · Issue #44925 · moby/moby · GitHub. Please post complete logs (e.g. journalctl -u docker.service) if the resolutions in that issue do not improve things for you.

Topic		Replies	Views
Failed to start Docker General raspberrypi	3	13673	May 8, 2020
Docker Broken After Update? General docker	0	74	June 18, 2024
Failed to start the docker on the raspberry pi General	1	437	October 5, 2023
Docker in Raspberry Pi General	10	13764	February 12, 2019
Unable to connect to docker daemon after apt-get upgrade on ubuntu based server General docker	0	2459	March 4, 2018

Apt upgrade on Raspberry Pi4 broke docker when it upgraded to 23.0.0

Related Topics