Issues (re)installing docker on Ubuntu 22.04

Docker Engine General
1

Hello,

today docker on my server crashed an i am not getting it running again.
Usually i have my docker running with docker compose. Today i edited my docker compose file and after running the docker-compose up -d command everything went to hell.

I tried setting it back up as discribed here but still having no luck.

moet@ubuntu-linux:/opt$ for pkg in docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc; do sudo apt-get remove $pkg; done
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »docker.io« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »docker-doc« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »docker-compose« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »docker-compose-v2« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »podman-docker« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »containerd« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
Paket »runc« ist nicht installiert, wird also auch nicht entfernt.
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.

moet@ubuntu-linux:/opt$ sudo apt-get update
OK:1 https://download.docker.com/linux/ubuntu jammy InRelease
OK:2 https://download.virtualbox.org/virtualbox/debian jammy InRelease                                       
OK:3 http://de.archive.ubuntu.com/ubuntu jammy InRelease                                                     
OK:4 http://de.archive.ubuntu.com/ubuntu jammy-updates InRelease                                             
OK:5 http://de.archive.ubuntu.com/ubuntu jammy-backports InRelease                   
OK:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
Paketlisten werden gelesen… Fertig

moet@ubuntu-linux:/opt$ sudo apt-get install ca-certificates curl gnupg
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
ca-certificates ist schon die neueste Version (20230311ubuntu0.22.04.1).
curl ist schon die neueste Version (7.81.0-1ubuntu1.14).
gnupg ist schon die neueste Version (2.2.27-3ubuntu2.1).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.

moet@ubuntu-linux:/opt$ sudo install -m 0755 -d /etc/apt/keyrings

moet@ubuntu-linux:/opt$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
Datei '/etc/apt/keyrings/docker.gpg' existiert bereits. Überschreiben (j/N)? j

moet@ubuntu-linux:/opt$ sudo chmod a+r /etc/apt/keyrings/docker.gpg

moet@ubuntu-linux:/opt$ echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(. /etc/os-release && echo "$VERSION_CODENAME") stable" | \
  sudo tee /etc/apt/sources.list.d/docker.list > /dev/null

moet@ubuntu-linux:/opt$ sudo apt-get update
OK:1 http://de.archive.ubuntu.com/ubuntu jammy InRelease
OK:2 http://de.archive.ubuntu.com/ubuntu jammy-updates InRelease                                                                                                                  
OK:3 https://download.docker.com/linux/ubuntu jammy InRelease                                                                                                                                     
OK:4 http://de.archive.ubuntu.com/ubuntu jammy-backports InRelease                                                                                                                                
OK:5 https://download.virtualbox.org/virtualbox/debian jammy InRelease                                                                        
OK:6 http://security.ubuntu.com/ubuntu jammy-security InRelease
Paketlisten werden gelesen… Fertig

moet@ubuntu-linux:/opt$ sudo apt-get install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
containerd.io ist schon die neueste Version (1.6.25-1).
docker-buildx-plugin ist schon die neueste Version (0.11.2-1~ubuntu.22.04~jammy).
docker-ce-cli ist schon die neueste Version (5:24.0.7-1~ubuntu.22.04~jammy).
docker-ce ist schon die neueste Version (5:24.0.7-1~ubuntu.22.04~jammy).
docker-compose-plugin ist schon die neueste Version (2.21.0-1~ubuntu.22.04~jammy).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.

moet@ubuntu-linux:/opt$ sudo docker run hello-world
Unable to find image 'hello-world:latest' locally
latest: Pulling from library/hello-world
719385e32844: Pulling fs layer 
docker: open /var/lib/docker/tmp/GetImageBlob3064665096: no such file or directory.
See 'docker run --help'.

Besides the last block it basically says, that all docker things are running on the newest version. But when i remove them again, as i did in the first code block, i get the message that they cant be removed because they are not installed…

i have no idea what to do.

Anyone who can help and shed some light on my problem?

Thank you in advance,
Moet

2

Your first code block is a cumbersome way for sudo apt-get remove -y docker.io docker-doc docker-compose docker-compose-v2 podman-docker containerd runc. It is used to remove packages provided by the os repositories.

Then you installed these packages from docker’s repos: sudo apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

So if you want to remove them, you need to run sudo apt-get remove -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

But I doubt that removing and reinstalling the package will result in any difference.

You could stop the docker service, move /var/lib/docker to a different location, then start the docker service again, so it will start with a fresh docker root directory,

Before you do anything, could you share the output of these commands:

sudo docker info
df -h
3 
Client: Docker Engine - Community
 Version:    24.0.7
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.11.2
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.21.0
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 20
  Running: 0
  Paused: 0
  Stopped: 20
 Images: 11
 Server Version: 24.0.7
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: systemd
 Cgroup Version: 2
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d8f198a4ed8892c764191ef7b3b06d8a2eeb5c7f
 runc version: v1.1.10-0-g18a0cb0
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
  cgroupns
 Kernel Version: 6.2.0-36-generic
 Operating System: Ubuntu 22.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 15.5GiB
 Name: ubuntu-linux
 ID: 73d54849-f2fa-4a1f-ab38-5a19b519af2f
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

tmpfs                        1,6G    3,4M  1,6G    1% /run
/dev/nvme0n1p3               916G     36G  834G    5% /
tmpfs                        7,8G       0  7,8G    0% /dev/shm
tmpfs                        5,0M    4,0K  5,0M    1% /run/lock
/dev/nvme0n1p2               512M    6,1M  506M    2% /boot/efi
tmpfs                        1,6G     72K  1,6G    1% /run/user/128
tmpfs                        1,6G     64K  1,6G    1% /run/user/1000

I do not have that directory…

Thanks for the effort.

4

var-lib
var-lib1452×139 12.5 KB

5

Your docker installation begs to differ :slight_smile:

Furthermore, we now know that the backing file system of that folder is ext4, so that the storage driver overlay2 is used, and there is no issue with space in the file system.

It could be a corruption in the docker root dir. At least that’s what I suspected and therefor suggested to rename the folder, and start with a fresh state (like it would be on a system where docker-ce was never installed before)

6

Now I see it. But it doesn’t make sense to me.

The output of docker info clearly shows that it uses this folder as docker root dir. When the folder does not exist, it should be created and pre-populated with the default settings.

BUT: i see that you make your post in Docker Desktop. So in this case docker-ce should be irrelevant (and preferably deleted again).

Please share the output of docker context ls (as normal user, without sudo).

I suspect a double installation: Docker Desktop + Docker CE.

7

Sorry, i might have posted in the wrong section. I am using it only terminal based… so no desktop i guess.

NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
8

Yep wrong category. I moved it to `Open Source Projects / Docker Engine"

There is no docker desktop installed, otherwise we would see an additional context.

Still there is one thing that doesn’t add up: Even though your docker daemon is running (otherwise docker info would have resulted in an error) and shows that /var/lib/docker is used, your screenshot shows that the folder is missing. I have no idea how this is even possible… It doesn’t make sense to me.

Can you share the content of cat /etc/docker/daemon.json. Note: the file does not exist by default.

9

It says that its not found.

Shouldn´t i get the right version using sudo apt-get install docker-ce docker-ce-cli containerd.io ?

Instead i get

Paketlisten werden gelesen… Fertig
Abhängigkeitsbaum wird aufgebaut… Fertig
Statusinformationen werden eingelesen… Fertig
containerd.io ist schon die neueste Version (1.6.25-1).
docker-ce-cli ist schon die neueste Version (5:24.0.7-1~ubuntu.22.04~jammy).
docker-ce ist schon die neueste Version (5:24.0.7-1~ubuntu.22.04~jammy).
0 aktualisiert, 0 neu installiert, 0 zu entfernen und 54 nicht aktualisiert.

its very weired.

10

It is okay if the file doesn’t exist. It just means the docker daemon runs with the default configuration.

I doubt that the packages are your problem. Yours should look like mine (just for Ubuntu 22.04 instead of 20.04)

me@docker:~$ dpkg -l | grep -E '(docker|containerd)'
ii  containerd.io                          1.6.24-1                          amd64        An open and reliable container runtime
ii  docker-buildx-plugin                   0.11.2-1~ubuntu.20.04~focal       amd64        Docker Buildx cli plugin.
ii  docker-ce                              5:24.0.7-1~ubuntu.20.04~focal     amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:24.0.7-1~ubuntu.20.04~focal     amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:24.0.7-1~ubuntu.20.04~focal     amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.21.0-1~ubuntu.20.04~focal       amd64        Docker Compose (V2) plugin for the Docker CLI.

The weird part is that your docker daemon is working, but the docker root dir it uses does not exist.

Did you modify the systemd unit?

Update: thank you, you made me aware there is a new version of containerd.io. Just installed it. So i am on 1.6.25-1 now as well.

11

They do look like yours.

Not to my knowlege… i am pretty new to linux so i try to use the basic configs. It all ran fine until earlier today…

12

Can you share the output of sudo stat /var/lib/docker?

13 
Datei: /var/lib/docker
 Größe: 4096            Blöcke: 8          EA Block: 4096   Verzeichnis
Gerät: 10303h/66307d    Inode: 50204875    Verknüpfungen: 12
Zugriff: (0710/drwx--x---)  Uid: (    0/    root)   Gid: (    0/    root)
Zugriff: 2023-11-26 18:08:50.024632112 +0100
Modifiziert: 2023-11-26 18:08:50.524632109 +0100
Geändert: 2023-11-26 18:08:50.524632109 +0100
Geburt: 2023-11-26 18:08:49.900632113 +0100
14

Actually it makes sense that it exists (as your docker daemon uses it).

Question is: why doesn’t your user see the folder? Could it be that apparmor or acls are preventing something in that folder?

This command should show that the docker-default profile is used and each container process actually uses it as well:

sudo aa-status | grep docker
15

it gives back docker-default

16

Only once?

Mine does so for each and every process that runs in containers.

So apparently none of the 20 containers docker info showed is actually running.

17

You might want to check the daemon logs for error messages: Read the daemon logs | Docker Docs

18

i trolled trough the logs and the last thing in there is this:

-- Boot e13b7971dc7b47f29e01cfa91c03d764 --
Nov 26 18:08:49 ubuntu-linux systemd[1]: Starting Docker Application Container Engine...
░░ Subject: A start job for unit docker.service has begun execution
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has begun execution.
░░ 
░░ The job identifier is 186.
Nov 26 18:08:49 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:49.906650587+01:00" level=info msg="Starting up"
Nov 26 18:08:49 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:49.910446759+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf"
Nov 26 18:08:50 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:50.039650852+01:00" level=info msg="Loading containers: start."
Nov 26 18:08:50 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:50.491414980+01:00" level=info msg="Loading containers: done."
Nov 26 18:08:50 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:50.527957171+01:00" level=info msg="Docker daemon" commit=311b9ff graphdriver=overlay2 version=24.0.7
Nov 26 18:08:50 ubuntu-linux dockerd[1274]: time="2023-11-26T18:08:50.528249145+01:00" level=info msg="Daemon has completed initialization"
Nov 26 18:08:50 ubuntu-linux systemd[1]: Started Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished successfully.
░░ 
░░ The job identifier is 186.

i Could also share the log from the time i crashed

19

Can you share the output of sudo docker ps -a --no-trunc?

20

Its comming up empty:

CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

Update:
It looks like i am able to run my docker-compose now