After upgraded to docker 23.0.0 unable to start

Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

docker version
Client: Docker Engine - Community
Version: 23.0.0
API version: 1.42
Go version: go1.19.5
Git commit: e92dd87
Built: Wed Feb 1 17:49:08 2023
OS/Arch: linux/amd64
Context: default

I was doing normal ubuntu upgrade using “apt update && apt upgrade -y”…my ubuntu was upgraded from “ubuntu.20.04~focal” to “Ubuntu 22.04.1 LTS” will it be the issue ?

Errors were encountered while processing:
docker-ce
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

Setting up docker-ce (5:23.0.0-1~ubuntu.20.04~focal) …
Job for docker.service failed because the control process exited with error code.
See “systemctl status docker.service” and “journalctl -xeu docker.service” for details.
invoke-rc.d: initscript docker, action “restart” failed.
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: activating (auto-restart) (Result: exit-code) since Sun 2023-02-05 13:43:58 UTC; 4ms ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Process: 5021 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=2)
Main PID: 5021 (code=exited, status=2)
CPU: 307ms
dpkg: error processing package docker-ce (–configure):
installed docker-ce package post-installation script subprocess returned error exit status 1
Errors were encountered while processing:
docker-ce

It’s hard to say for sure as you have not provided daemon logs (journalctl -u docker.service), but you are most likely missing apparmor: sudo apt install apparmor

1 Like

I ran into the same issue.
In my case it was an issue with the daemon.json file. It appears that the update broke the file. All I had to do was delete an extra } character and restart the service

unable to configure the Docker daemon with file /etc/docker/daemon.json: invalid character '}' after top-level value

I’m going to piggyback on this rather than create a new topic as I am having the same issue and the logs are not giving me any clues; I’d really appreciate some help.

Ubuntu and docker versions are exactly the same as OP has stated.
I have 3 Ubuntu (VM) machines running docker and the other two have updated to the latest version without issue.
I have checked based on @neersighted suggestion and apparmor is installed.

❯ sudo apt list apparmor
Listing... Done
apparmor/jammy-updates,now 3.0.4-2ubuntu2.1 amd64 [installed,automatic]

I don’t have a daemon.json file so i doubt that is the problem.

I’ve included the output of journalctl -eu docker.service below:

Feb 06 13:56:21 dockerhost2 dockerd[325692]: main.(*DaemonCli).start(0xc000c3d830, 0xc0007771f0)
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:200 +0x9f6
Feb 06 13:56:21 dockerhost2 dockerd[325692]: main.runDaemon(...)
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:14
Feb 06 13:56:21 dockerhost2 dockerd[325692]: main.newDaemonCommand.func1(0xc0000f6300?, {0xc000c3d7d0?, 0x3?, 0x3?})
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:38 +0x5e
Feb 06 13:56:21 dockerhost2 dockerd[325692]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(>
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/com>
Feb 06 13:56:21 dockerhost2 dockerd[325692]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC>
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/com>
Feb 06 13:56:21 dockerhost2 dockerd[325692]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(>
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/com>
Feb 06 13:56:21 dockerhost2 dockerd[325692]: main.main()
Feb 06 13:56:21 dockerhost2 dockerd[325692]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:102 +0x15d
Feb 06 13:56:21 dockerhost2 systemd[1]: docker.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Feb 06 13:56:21 dockerhost2 systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 06 13:56:21 dockerhost2 systemd[1]: Failed to start Docker Application Container Engine.
Feb 06 13:56:23 dockerhost2 systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Feb 06 13:56:23 dockerhost2 systemd[1]: Stopped Docker Application Container Engine.
Feb 06 13:56:23 dockerhost2 systemd[1]: docker.service: Start request repeated too quickly.
Feb 06 13:56:23 dockerhost2 systemd[1]: docker.service: Failed with result 'exit-code'.
Feb 06 13:56:23 dockerhost2 systemd[1]: Failed to start Docker Application Container Engine.

I have tried googling but have not found anything that gives me any ideas.
I have also tried (as suggested somewhere else) to run sudo dockerd --debug to see if that shows anything but I don’t think that is showing any errors.

I have restored my VM from the previous backup and tried the process again with the same result.
I have restored again so that everything is working but I am in a position where I can’t update this server now. If worst comes to worst I could reinstall ubuntu and reimport my docker containers but I’d rather avoid that if at all possible.

Happy to include anything else that could assist with any troubleshooting.

Thanks in advance

You have not provided enough log output to determine what the failure is, but there is definitely a panic in there. The output of dockerd --debug would be helpful – that is basically the same output systemd generates. I suspect it is failing and you don’t know what to look for.

Could you please provide journalctl -u docker.service, which will provide the complete logs for all runs of dockerd since boot? I’m guessing this is panic: interface {} is nil in libnetwork/drivers/ipvlan · Issue #44925 · moby/moby · GitHub, since that regression is the only panic on startup we’ve seen, but it’s also possible you’ve hit something novel. We won’t know without complete logs.

thanks so much @neersighted

I soon as I posted this I saw the other post and the link to the github issue and yes I am seeing that in the output of sudo dockerd --debug

I do indeed have an ipvlan so I will remove that first and then attempt the upgrade as suggested in the github issue.
I will report back.

Yes, I can confirm this fixed my issue (with a bit of post-update tidying up).

I removed everything under /var/lib/docker/network
Docker update (via apt) then went through successfully.

I did have an issue when using docker compose to bring everything back up that it complained it couldn’t find a network so I issued a docker system prune, restarted the docker service and then docker compose up -d and everything came back to life (I create my ipvlan as part of my docker-compose file).

Apologies for any additional noise I created and thanks so much for such a quick response.

No problem! Just remember to include complete logs whenever you ask for help in the future, and we can reduce the round trips. Good job investigating the issue and following the mitigation! :smile:

Hi All!

Just to inform you, I had the ‘same’ problem (I mean, not working Docker after the 23.0 upgrade), but because the depriciated graph definition in the daemon.json.

After changed that to data-root, my problem have solved.

1 Like

confirmed latest docker release solved this issue

Your solution works! Thank you.

1 Like