My docker container and images are not visible in sudo docker images and sudo docker ps -l . The docker installation on my system requires sudo while running any docker commands.
I can see with df -h that the same memory is being occupied on my drives, but even with sudo docker info it says container 0 and images 0.
Also every time there is a typo, it gives this warning:
WARNING:root:could not open file '/etc/apt/sources.list.d/docker.list'
WARNING:root:could not open file '/etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list'
WARNING:root:could not open file '/etc/apt/sources.list.d/microsoft-prod.list'
I have checked most posts on forums that restarting docker with systemctl bring back all the images, but not in my case. Can someone point me to what issue there might be?
Edit1:
seems /etc/apt/sources.list.d didnât have root permission so doing chmod +x fixed it.
Please check if you have installed a version from the snap repo and a version from a deb repo at the same time!
snap list docker should display:
error: no matching snaps installed
dpkg -l |grep docker should display something like this:
ii docker-ce 5:20.10.16~3-0~ubuntu-focal amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:20.10.16~3-0~ubuntu-focal amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:20.10.16~3-0~ubuntu-focal amd64 Rootless support for Docker.
ii docker-compose-plugin 2.5.0~ubuntu-focal amd64 Docker Compose (V2) plugin for the Docker CLI.
ii docker-scan-plugin 0.17.0~ubuntu-focal amd64 Docker scan cli plugin.
ko
Make sure to not use the snap package, and if a deb package is installed, that itâs the packages you see in the output I shared and not the package from ubuntuâs own repositories!
$> snap list docker
error: no matching snaps installed
$> dpkg -l |grep docker
ii docker-ce 5:20.10.17~3-0~ubuntu-bionic amd64 Docker: the open-source application container engine
ii docker-ce-cli 5:20.10.17~3-0~ubuntu-bionic amd64 Docker CLI: the open-source application container engine
ii docker-ce-rootless-extras 5:20.10.17~3-0~ubuntu-bionic amd64 Rootless support for Docker.
ii docker-compose-plugin 2.6.0~ubuntu-bionic amd64 Docker Compose (V2) plugin for the Docker CLI.
ii docker-scan-plugin 0.17.0~ubuntu-bionic amd64 Docker scan cli plugin.
Did you change the storage driver recently (in /etc/docker/daemon.json, the file does not exist by default)?
The warnings you posted earlier are related to apt. They should not relate in any way with what you experience with docker. Though, it might still be an indication that something is wrong with your system.
Normaly I would recommend to remove and re-install the docker-ce* packages. But as you seem to have problems with apt, I am not sure if you will be able to do so.
Just to be sure: those warning are on the host and not inside one of your containers.
Did you change the storage driver recently (in /etc/docker/daemon.json, the file does not exist by default)?
Previously I was trying to change the containers save location from root disk to other disk. However it failed, and I did a purge of docker installations and files and started now with a fresh install of docker. So the /etc/docker/daemon.json does not exist now.
The warnings are now gone, I had to change permissions with chmod +x, those warning are not coming anymore.
Normaly I would recommend to remove and re-install the docker-ce* packages. But as you seem to have problems with apt, I am not sure if you will be able to do so
Will removing and reinstalling docker-ce hurt the images which were saved previously? I donât want to loose them.
Do you remember what the storage driver was before you reinstalled Docker? Even if the data root is the original again, when the storage driver changes, it also changes the file structure in the data root folder so nothing will be found from the old data.
Check the storage driver in the output of docker info and list the folders in the docker data root to see if you can find folders indicating that you had a different storage driver before. âoverlay2â is the most popular recently. In case of overlay2, the folders should look like this:
I did not change the storage-driver, I recall from previous installation that overlay-2 was not supported/recognized when I was trying to change docker data dir to an external HDD.
By this I meant, I kept checking while committing new images to docker and my storage is occupying the same space as with all the docker data
Could you show that too? If you could not see the images after a reboot, that probably means Docker started with a different configuration than it had before.
$> journalctl -e -u docker
Jun 13 11:53:47 p1 dockerd[3967]: time="2022-06-13T11:53:47.643294261+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <n
Jun 13 11:53:47 p1 dockerd[3967]: time="2022-06-13T11:53:47.643310150+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.090769356+05:30" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246679808+05:30" level=warning msg="Your kernel does not support swap memory limit"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246726871+05:30" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246756382+05:30" level=warning msg="Your kernel does not support cgroup blkio weight"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246778671+05:30" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.247155651+05:30" level=info msg="Loading containers: start."
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.487642556+05:30" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to se
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.594966797+05:30" level=info msg="Loading containers: done."
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.712421062+05:30" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.712512893+05:30" level=info msg="Daemon has completed initialization"
Jun 13 11:53:48 p1 systemd[1]: Started Docker Application Container Engine.
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.816067736+05:30" level=info msg="API listen on /var/run/docker.sock"
Jun 13 12:06:48 p1 dockerd[3967]: time="2022-06-13T12:06:48.055247312+05:30" level=info msg="Processing signal 'terminated'"
Jun 13 12:06:48 p1 dockerd[3967]: time="2022-06-13T12:06:48.056853313+05:30" level=info msg="Daemon shutdown complete"
Jun 13 12:06:48 p1 systemd[1]: Stopping Docker Application Container Engine...
Jun 13 12:06:48 p1 systemd[1]: Stopped Docker Application Container Engine.
-- Reboot --
Jun 13 12:07:58 p1 systemd[1]: Starting Docker Application Container Engine...
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.196317332+05:30" level=info msg="Starting up"
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.208226903+05:30" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336402492+05:30" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336457555+05:30" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336494805+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <n
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336514382+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347143750+05:30" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347188374+05:30" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347221686+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock <nil> 0 <nil>}] <nil> <n
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347241944+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.087963144+05:30" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414072007+05:30" level=warning msg="Your kernel does not support swap memory limit"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414127250+05:30" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414145605+05:30" level=warning msg="Your kernel does not support cgroup blkio weight"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414162196+05:30" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414519897+05:30" level=info msg="Loading containers: start."
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.927428930+05:30" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to se
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.113354198+05:30" level=info msg="Loading containers: done."
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.614318453+05:30" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.636970284+05:30" level=info msg="Daemon has completed initialization"
Jun 13 12:08:03 p1 systemd[1]: Started Docker Application Container Engine.
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.900954146+05:30" level=info msg="API listen on /var/run/docker.sock"
Jun 13 12:14:37 p1 dockerd[1477]: time="2022-06-13T12:14:37.005840542+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:14:37 p1 dockerd[1477]: time="2022-06-13T12:14:37.005946316+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:15:16 p1 dockerd[1477]: time="2022-06-13T12:15:16.036322897+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:15:16 p1 dockerd[1477]: time="2022-06-13T12:15:16.036388580+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:17:43 p1 dockerd[1477]: time="2022-06-13T12:17:43.126973542+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:17:43 p1 dockerd[1477]: time="2022-06-13T12:17:43.127038938+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:18:36 p1 dockerd[1477]: time="2022-06-13T12:18:36.947497044+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:18:36 p1 dockerd[1477]: time="2022-06-13T12:18:36.947574740+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:50:00 p1 dockerd[1477]: time="2022-06-13T12:50:00.492019654+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:50:00 p1 dockerd[1477]: time="2022-06-13T12:50:00.492080609+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Accoring Docker storage drivers | Docker Docs the overlay2 storage driver only supports xfs with ftype=1 and ext4. extfs is not considered as ext4, is it?
Does this mean you had the data on the old location, changed the data dir without moving the data from the old location to the new location? This might explain why old data is missing. Though, I guess your problem is that you forcefully configured overlay2 with a backed filesystem that doesnât support it properly. So even if you move the files without fixing this first, it will high likely be still broken.
You need to format your external hd to ext4 and then move existing data to the new location to make it work.
Though: its generaly not recommendable to have the data-root on a removable mediumâŠ
I had tried to move my docker root to another location, but it did not work. So I completely deleted docker and all files. With the new installation of docker I will keep the docker root at same location as it is default assigned.
Now I created new images and committed them. While committing, I kept in check how much of my storage is occupied. After committing all changes last week, I shutdown my system and when I restarted yesterday I am not able to see any images with sudo docker images and sudo docker ps -l . However the same amount of storage space is occupied as when I could see the images.
I can say this because I had multiple images on which I worked on and the data was around 28GiB and it would have been 28GiB free if all the images were deleted in some case.
$> sudo docker login
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/ashish/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store
Login Succeeded
Even after a successful login the journalctl reports level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
update:
I tried to access my image which I had committed before shutting down my system:
$> sudo docker run -it ashish_toolchain_dep2:v2 bash
Unable to find image 'ashish_toolchain_dep2:v2' locally
docker: Error response from daemon: pull access denied for ashish_toolchain_dep2, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.
Your filesystem is corrupted so even if most of the data is in the docker data root, some files are missing, not readable or have the wrong content so Docker donât know how those files are related.
You have multiple contexts and somehow it was changed after the reboot so the data is there, but the other context stores data in an other directory. It can happen when you install rootless docker or use a new user namespace. The latter is what we would have seen in your folder structure, so that is not the case.
Recommendations:
Image metadata is in /var/lib/docker/image/overlay2. Check the content of ârepositories.jsonâ in that folder.
See what you can find in /var/lib/docker/image/overlay2/imagedb/content/sha256. Here are the image metadata fjson iles
Check the content of /var/lib/docker/image/overlay2/layerdb/sha256/ these are the folders for some layer data.
Try to create a new image and check if it works or the filesystem is not usable at all.
The above steps will not solve anything, but you may notice something, for example that everything is empty. In case of filesystem corruption, you need to repair the filesystem or replace the disk. I donât want to go into details about filesystem corruptions, but you can search for âfsckâ
In case you have multiple contexts, you can find them by running docker context ls and switch to default by running docker context use default
Yes, everything was empty. As a test, I pulled the hello-world image and it worked when I run it. I pulled another image made some changes, committed it and it worked. So it seems the filesystem is not corrupted. I am also using other stuff on my machine, and if the fs was corrupted I could have seen some effects over them as well.
I also checked the context and it was set to default. So nothing on this lead as well.
Anyway I decided to do a fsck on root partition, and this method required to make some changes to /etc/fstab .
I found a statement there that I had made changes in previous installation of docker to bind mount to a dir on my other drive. I deleted that statement, rebooted and all my images were visible with sudo docker images .
I am aware that actions of the past come to haunt you, but this was very scary
