Ubuntu 22.04 and 5:23.0.0-1 now fails to start - please suggest debug

markhannon · February 5, 2023, 2:39am

Hi,

In the last few days I performed a regular dist-upgrade and now docker-ce fails to start. I have removed, purged, re-installed etc and have not found anything.

The upgrade fails like this:

Setting up docker-ce (5:23.0.0-1~ubuntu.22.04~jammy) ...
Job for docker.service failed because the control process exited with error code.
See "systemctl status docker.service" and "journalctl -xeu docker.service" for details.
invoke-rc.d: initscript docker, action "start" failed.
● docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: activating (auto-restart) (Result: exit-code) since Sun 2023-02-05 13:33:54 AEDT; 6ms ago
TriggeredBy: ● docker.socket
       Docs: https://docs.docker.com
    Process: 1509861 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)
   Main PID: 1509861 (code=exited, status=2)
        CPU: 281ms

Feb 05 13:33:54 hal systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 05 13:33:54 hal systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 05 13:33:54 hal systemd[1]: Failed to start Docker Application Container Engine.
dpkg: error processing package docker-ce (--configure):
 installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
 docker-ce
E: Sub-process /usr/bin/dpkg returned an error code (1)

Checking the journal doesn’t give me any clues except a panic message about interfaces:

Feb 05 13:33:59 hal dockerd[1510161]: panic: interface conversion: interface {} is nil, not string
Feb 05 13:33:59 hal dockerd[1510161]: goroutine 1 [running]:
Feb 05 13:33:59 hal dockerd[1510161]: github.com/docker/docker/libnetwork/drivers/ipvlan.(*configuration).UnmarshalJSON(0xc000e73830, {0xc000bcc000, 0xee, 0xee})
Feb 05 13:33:59 hal dockerd[1510161]:         /go/src/github.com/docker/docker/libnetwork/drivers/ipvlan/ipvlan_store.go:196 +0x40c

Feb 05 13:33:59 hal dockerd[1510161]: main.main()
Feb 05 13:33:59 hal dockerd[1510161]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:102 +0x15d
Feb 05 13:33:59 hal systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT

The status is also not helpful:

mark@hal:/etc/docker sudo systemctl status docker.service                                                                                                                                                 [13:36:30]
× docker.service - Docker Application Container Engine
     Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Sun 2023-02-05 13:34:01 AEDT; 2min 29s ago
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 1510161 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)
   Main PID: 1510161 (code=exited, status=2)
        CPU: 196ms

Feb 05 13:33:59 hal systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 05 13:33:59 hal systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 05 13:33:59 hal systemd[1]: Failed to start Docker Application Container Engine.
Feb 05 13:34:01 hal systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Feb 05 13:34:01 hal systemd[1]: Stopped Docker Application Container Engine.
Feb 05 13:34:01 hal systemd[1]: docker.service: Start request repeated too quickly.
Feb 05 13:34:01 hal systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 05 13:34:01 hal systemd[1]: Failed to start Docker Application Container Engine.

The versions installed are:

mark@hal:/etc/docker dpkg -l | egrep "docker|container"                                                                                                                                                   [13:37:27]
ii  containerd.io                              1.6.16-1                                amd64        An open and reliable container runtime
ii  docker-buildx-plugin                       0.10.2-1~ubuntu.22.04~jammy             amd64        Docker Buildx cli plugin.
iF  docker-ce                                  5:23.0.0-1~ubuntu.22.04~jammy           amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:23.0.0-1~ubuntu.22.04~jammy           amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras                  5:23.0.0-1~ubuntu.22.04~jammy           amd64        Rootless support for Docker.
ii  docker-compose-plugin                      2.15.1-1~ubuntu.22.04~jammy             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  docker-scan-plugin                         0.23.0~ubuntu-jammy                     amd64        Docker scan cli plugin.
ii  libnss-mymachines:amd64                    249.11-0ubuntu3.6                       amd64        nss module to resolve hostnames for local container instances
ii  systemd-container                          249.11-0ubuntu3.6                       amd64        systemd container/nspawn tools

I have reverted to iptables-legacy.

Any other ideas to debug this behaviour?

geckonz · February 5, 2023, 7:58am

I am finding same issue but on Debian 11.6. Same version of Docker though. I am also using ipvlan driver and suspect this is contributing.
Running journalctl for me results in same panic message.

Feb 05 10:16:33 wilbur dockerd[4622]: time=“2023-02-05T10:16:33.206756897+13:00” level=info msg="Loading containers: st>
Feb 05 10:16:33 wilbur dockerd[4622]: panic: interface conversion: interface {} is nil, not string
Feb 05 10:16:33 wilbur dockerd[4622]: goroutine 1 [running]:
Feb 05 10:16:33 wilbur dockerd[4622]: github.com/docker/docker/libnetwork/drivers/ipvlan.(*configuration).UnmarshalJSON>
Feb 05 10:16:33 wilbur dockerd[4622]: /go/src/github.com/docker/docker/libnetwork/drivers/ipvlan/ipvlan_store.g>
Feb 05 10:16:33 wilbur dockerd[4622]: encoding/json.(*decodeState).object(0xc000649b90, {0x55f6023c9660?, 0xc000649b00?>
Feb 05 10:16:33 wilbur dockerd[4622]: /usr/local/go/src/encoding/json/decode.go:613 +0x6ef

icet · February 5, 2023, 9:04am

Seeing the same in Ubuntu 22 with an ipvlan config. Confirmed downgrading to 5:20.10.23 restores functionality.

foreverlate88 · February 5, 2023, 1:55pm

gosh same issue, no idea how to fix

foreverlate88 · February 5, 2023, 4:12pm

thx for the hint
downgraded to 5:20.10.23~3-0~ubuntu-jammy worked for me at least worked

meyay · February 5, 2023, 5:53pm

This should re-install the docker-ce 20.10.23:

sudo apt install --allow-downgrades -y docker-ce=$(apt-cache madison docker-ce | awk '/20.10.23/{print $3}')

Then mark the package so it’s held back when upgrading packages:

sudo apt-mark hold docker-ce

jsantalo · February 5, 2023, 6:01pm

Worked for me! Many thanks @meyay !
Now, let´s find out the source of the problem on why the 23.0.0 upgrade fails …

markhannon · February 5, 2023, 8:44pm

thanks, worked a treat

geckonz · February 5, 2023, 9:10pm

Downgrade worked for me on Debian too, although I also downgraded the following additional packages - call me paranoid

sudo apt install --allow-downgrades docker-ce=5:20.10.23~3-0~debian-bullseye
sudo apt install --allow-downgrades docker-ce-cli=5:20.10.23~3-0~debian-bullseye
sudo apt install --allow-downgrades docker-ce-rootless-extras=5:20.10.23~3-0~debian-bullseye

sudo apt-mark hold docker-ce docker-ce-cli docker-ce-rootless-extras

jblachly · February 5, 2023, 10:10pm

Same problem, but it broke my automated packer builds

What a disaster

meyay · February 5, 2023, 10:24pm

Thus the breakage happend on a fresh system…

I just created a kvm vm based on a recent ubuntu 22.04 cloudinit image and installed docker using the convinience installe script: I expected to have a brocken docker engine as well, but instead it worked.

I am curious if the breakage is related to settings in /etc/docker/daemon.json or modifiations of the systemd unit for docker.

jblachly · February 6, 2023, 12:04am

I just created a kvm vm based on a recent ubuntu 22.04 cloudinit image and installed docker using the convinience installe script: I expected to have a brocken docker engine as well, but instead it worked.

Based on this, I examined my build scripts more closely.

Prior to installation of Docker, I was installing awscli and amazon-ecr-credential-helper; the latter forced installation of the universe docker.io package.

When I manually repeated installation steps but omitted amazon-ecr-credential-helper (thus ensuring the Docker CE engine 23.0 was installed on a never-seen-docker-system, it succeeded.

So, this definitely is an upgrade issue.

foreverlate88 · February 6, 2023, 1:53am

I cloned my docker VM to troubleshoot. Apparently it require me to remove /var/lib/docker to do a reinstall else I wont be able to install 23.0 and stuck with the same error, i guess is compatibility issue when upgrading

suburbanau · February 6, 2023, 3:26am

I can report also had the same issue on Ubuntu 22.04. Was able to recover by running the restoration / rollback command mentioned above.

System has been stable with Docker for the last year.

Hopefully this can get fixed soon whatever is causing the upgrade issue for existing systems.

suburbanau · February 6, 2023, 3:32am

Worked well for me! Thank you.

neersighted · February 6, 2023, 7:02am

panic: interface {} is nil in libnetwork/drivers/ipvlan · Issue #44925 · moby/moby · GitHub is the tracking issue for this regression.

For others here who are having issues with 23.0.0, no one can help you without logs from the failed daemon start at the very least. If you have something that does not look like the issue I linked above, please post logs here, or if you comfortable creating a well-formatted issue, on the moby/moby issue tracker.

en666 · February 9, 2023, 12:32pm

Downgrading docker-ce and docker-ce-cli versions did the trick for me on Ubuntu. Detailed steps:

List how many version apt-get has: apt-cache policy docker-ce

docker-ce:
  Installed: 5:23.0.0-1~ubuntu.18.04~bionic
  Candidate: 5:23.0.0-1~ubuntu.18.04~bionic
  Version table:
 *** 5:23.0.0-1~ubuntu.18.04~bionic 500
        500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
      5:20.10.23~3-0~ubuntu-bionic 500
        500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
        100 /var/lib/dpkg/status
     5:20.10.22~3-0~ubuntu-bionic 500
        500 https://download.docker.com/linux/ubuntu bionic/stable amd64 Packages
     5:20.10.21~3-0~ubuntu-bionic 500

Say you choose the latest you know it was working, in my case “5:20.10.23~3-0~ubuntu-bionic”, launch the following to re-install docker with the specified version:

DOCKER_VERSION=5:20.10.23~3-0~ubuntu-bionic
sudo apt-get install docker-ce=$DOCKER_VERSION docker-ce-cli=$DOCKER_VERSION containerd.io docker-buildx-plugin docker-compose-plugin

Done, you should be able to run docker run hello-world (or sudo docker run hello-world if your user hasn’t got the ‘docker’ group)

neersighted · February 10, 2023, 7:30pm

23.0.1 fixes the regressions discussed in this thread, for what it is worth.

markhannon · February 10, 2023, 8:42pm

Thanks. I’ll try that

Regards, Mark

markhannon · February 10, 2023, 11:28pm

Tried and it works. Thanks

Topic		Replies	Views
Install docker-ce failed on Ubuntu 16.04.4 General docker	1	4892	March 19, 2018
Docker.service fails to start after upgrade from 24.0.2 to 24.0.3 General docker	2	1574	February 10, 2024
Cannot install docker on ubuntu 22.04.2 General docker , docker-compose , ubuntu	2	2565	May 9, 2023
Installation Problems on Fresh Ubuntu General	3	1037	November 29, 2018
Docker (-ce) not starting anymore after Ubuntu patching. (panic:interface conversion) General docker	4	1124	February 10, 2023

Ubuntu 22.04 and 5:23.0.0-1 now fails to start - please suggest debug

Related topics