Docker images and container lost not visible WARNING:root:could not open file '/etc/apt/sources.list.d/docker.list'

OS: Ubuntu 18.04 LTS
Docker 20.10.17

My docker container and images are not visible in sudo docker images and sudo docker ps -l . The docker installation on my system requires sudo while running any docker commands.

I can see with df -h that the same memory is being occupied on my drives, but even with sudo docker info it says container 0 and images 0.

Also every time there is a typo, it gives this warning:

WARNING:root:could not open file '/etc/apt/sources.list.d/docker.list'

WARNING:root:could not open file '/etc/apt/sources.list.d/ubuntu-toolchain-r-ubuntu-test-bionic.list'

WARNING:root:could not open file '/etc/apt/sources.list.d/microsoft-prod.list'

I have checked most posts on forums that restarting docker with systemctl bring back all the images, but not in my case. Can someone point me to what issue there might be?

Edit1:
seems /etc/apt/sources.list.d didn’t have root permission so doing chmod +x fixed it.

However I still can’t see the images

Please check if you have installed a version from the snap repo and a version from a deb repo at the same time!

snap list docker should display:

error: no matching snaps installed

dpkg -l |grep docker should display something like this:

ii  docker-ce                              5:20.10.16~3-0~ubuntu-focal       amd64        Docker: the open-source application container engine
ii  docker-ce-cli                          5:20.10.16~3-0~ubuntu-focal       amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras              5:20.10.16~3-0~ubuntu-focal       amd64        Rootless support for Docker.
ii  docker-compose-plugin                  2.5.0~ubuntu-focal                amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  docker-scan-plugin                     0.17.0~ubuntu-focal               amd64        Docker scan cli plugin.
ko

Make sure to not use the snap package, and if a deb package is installed, that it’s the packages you see in the output I shared and not the package from ubuntu’s own repositories!

Ok I checked as you mentioned:

$> snap list docker
error: no matching snaps installed
$> dpkg -l |grep docker
ii  docker-ce                                  5:20.10.17~3-0~ubuntu-bionic                    amd64        Docker: the open-source application container engine
ii  docker-ce-cli                              5:20.10.17~3-0~ubuntu-bionic                    amd64        Docker CLI: the open-source application container engine
ii  docker-ce-rootless-extras                  5:20.10.17~3-0~ubuntu-bionic                    amd64        Rootless support for Docker.
ii  docker-compose-plugin                      2.6.0~ubuntu-bionic                             amd64        Docker Compose (V2) plugin for the Docker CLI.
ii  docker-scan-plugin                         0.17.0~ubuntu-bionic                            amd64        Docker scan cli plugin.

Did you change the storage driver recently (in /etc/docker/daemon.json, the file does not exist by default)?

The warnings you posted earlier are related to apt. They should not relate in any way with what you experience with docker. Though, it might still be an indication that something is wrong with your system.

Normaly I would recommend to remove and re-install the docker-ce* packages. But as you seem to have problems with apt, I am not sure if you will be able to do so.

Just to be sure: those warning are on the host and not inside one of your containers.

Did you change the storage driver recently (in /etc/docker/daemon.json, the file does not exist by default)?

Previously I was trying to change the containers save location from root disk to other disk. However it failed, and I did a purge of docker installations and files and started now with a fresh install of docker. So the /etc/docker/daemon.json does not exist now.

The warnings are now gone, I had to change permissions with chmod +x, those warning are not coming anymore.

Normaly I would recommend to remove and re-install the docker-ce* packages. But as you seem to have problems with apt, I am not sure if you will be able to do so

Will removing and reinstalling docker-ce hurt the images which were saved previously? I don’t want to loose them.

Yes warning were on the host.

Do you remember what the storage driver was before you reinstalled Docker? Even if the data root is the original again, when the storage driver changes, it also changes the file structure in the data root folder so nothing will be found from the old data.

Check the storage driver in the output of docker info and list the folders in the docker data root to see if you can find folders indicating that you had a different storage driver before. “overlay2” is the most popular recently. In case of overlay2, the folders should look like this:

root@ta-lteg:/home/takacsakos# ls -la /var/lib/docker/
total 224
drwx--x---  15 root root     4096 jún   13 09:01 .
drwxr-xr-x  97 root root     4096 máj   26 11:15 ..
drwx------   2 root root     4096 máj    1  2020 builder
drwx--x--x   5 root root     4096 szept  6  2021 buildkit
drwx--x---   3 root root     4096 jún    9 13:40 containers
drwx------   3 root root     4096 máj    1  2020 image
drwxr-x---   3 root root     4096 máj    1  2020 network
-rwxr-xr-x   1 root root     1473 dec   14  2020 nuke-graph-directory.sh
drwx--x--- 576 root root   155648 jún   13 09:01 overlay2
drwx------   4 root root     4096 máj    1  2020 plugins
drwx------   2 root root     4096 jún   13 09:01 runtimes
drwx------   2 root root     4096 dec   27 21:29 swarm
drwx------   2 root root     4096 jún   13 09:01 tmp
drwx------   2 root root     4096 máj    1  2020 trust
drwx-----x  94 root root    12288 jún   13 09:01 volumes

If you can share the output of docker info and the list of folders in the data root, I will try to tell you what you can do.

1 Like

update:

Removing Docker will not delete the data (I tried it), but I don’t think it will help to find the old images.

update 2:

I wanted to actually update my previous message… :slight_smile:

$> sudo docker info
Client:
 Context:    default
 Debug Mode: false
 Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.8.2-docker)
  compose: Docker Compose (Docker Inc., v2.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 20.10.17
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 10c12954828e7c7c9b6e0ea9b0c02b01407d3ae1
 runc version: v1.1.2-0-ga916309
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.4.0-113-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 5.799GiB
 Name: p1
 ID: 25IR:57EE:Q3JM:4DRJ:YKER:6JKG:A7N4:5SMQ:IX6W:OHWR:Y7CJ:I3WD
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Username: ashishmagar
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

I did not change the storage-driver, I recall from previous installation that overlay-2 was not supported/recognized when I was trying to change docker data dir to an external HDD.

By this I meant, I kept checking while committing new images to docker and my storage is occupying the same space as with all the docker data

It shouldn’t matter.

So you use docker images, run containers and commit containers, but you can’t see those either?

Ahh… I completely missed it to mention.

I did a reboot on my system, and I am not able to see the images since then. I had committed all the changes to the images before the reboot

Could you show that too? If you could not see the images after a reboot, that probably means Docker started with a different configuration than it had before.

You should also check th system logs.

journalctl -e -u docker
sudo ls /var/lib/docker/
buildkit  containers  image  network  overlay2	plugins  runtimes  swarm  tmp  trust  volumes
$> journalctl -e -u docker
Jun 13 11:53:47 p1 dockerd[3967]: time="2022-06-13T11:53:47.643294261+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <n
Jun 13 11:53:47 p1 dockerd[3967]: time="2022-06-13T11:53:47.643310150+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.090769356+05:30" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246679808+05:30" level=warning msg="Your kernel does not support swap memory limit"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246726871+05:30" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246756382+05:30" level=warning msg="Your kernel does not support cgroup blkio weight"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.246778671+05:30" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.247155651+05:30" level=info msg="Loading containers: start."
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.487642556+05:30" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to se
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.594966797+05:30" level=info msg="Loading containers: done."
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.712421062+05:30" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.712512893+05:30" level=info msg="Daemon has completed initialization"
Jun 13 11:53:48 p1 systemd[1]: Started Docker Application Container Engine.
Jun 13 11:53:48 p1 dockerd[3967]: time="2022-06-13T11:53:48.816067736+05:30" level=info msg="API listen on /var/run/docker.sock"
Jun 13 12:06:48 p1 dockerd[3967]: time="2022-06-13T12:06:48.055247312+05:30" level=info msg="Processing signal 'terminated'"
Jun 13 12:06:48 p1 dockerd[3967]: time="2022-06-13T12:06:48.056853313+05:30" level=info msg="Daemon shutdown complete"
Jun 13 12:06:48 p1 systemd[1]: Stopping Docker Application Container Engine...
Jun 13 12:06:48 p1 systemd[1]: Stopped Docker Application Container Engine.
-- Reboot --
Jun 13 12:07:58 p1 systemd[1]: Starting Docker Application Container Engine...
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.196317332+05:30" level=info msg="Starting up"
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.208226903+05:30" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336402492+05:30" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336457555+05:30" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336494805+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <n
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.336514382+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347143750+05:30" level=info msg="parsed scheme: \"unix\"" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347188374+05:30" level=info msg="scheme \"unix\" not registered, fallback to default scheme" module=grpc
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347221686+05:30" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock  <nil> 0 <nil>}] <nil> <n
Jun 13 12:08:01 p1 dockerd[1477]: time="2022-06-13T12:08:01.347241944+05:30" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.087963144+05:30" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414072007+05:30" level=warning msg="Your kernel does not support swap memory limit"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414127250+05:30" level=warning msg="Your kernel does not support CPU realtime scheduler"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414145605+05:30" level=warning msg="Your kernel does not support cgroup blkio weight"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414162196+05:30" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.414519897+05:30" level=info msg="Loading containers: start."
Jun 13 12:08:02 p1 dockerd[1477]: time="2022-06-13T12:08:02.927428930+05:30" level=info msg="Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to se
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.113354198+05:30" level=info msg="Loading containers: done."
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.614318453+05:30" level=info msg="Docker daemon" commit=a89b842 graphdriver(s)=overlay2 version=20.10.17
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.636970284+05:30" level=info msg="Daemon has completed initialization"
Jun 13 12:08:03 p1 systemd[1]: Started Docker Application Container Engine.
Jun 13 12:08:03 p1 dockerd[1477]: time="2022-06-13T12:08:03.900954146+05:30" level=info msg="API listen on /var/run/docker.sock"
Jun 13 12:14:37 p1 dockerd[1477]: time="2022-06-13T12:14:37.005840542+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:14:37 p1 dockerd[1477]: time="2022-06-13T12:14:37.005946316+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:15:16 p1 dockerd[1477]: time="2022-06-13T12:15:16.036322897+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:15:16 p1 dockerd[1477]: time="2022-06-13T12:15:16.036388580+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:17:43 p1 dockerd[1477]: time="2022-06-13T12:17:43.126973542+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:17:43 p1 dockerd[1477]: time="2022-06-13T12:17:43.127038938+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:18:36 p1 dockerd[1477]: time="2022-06-13T12:18:36.947497044+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:18:36 p1 dockerd[1477]: time="2022-06-13T12:18:36.947574740+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"
Jun 13 12:50:00 p1 dockerd[1477]: time="2022-06-13T12:50:00.492019654+05:30" level=error msg="Not continuing with pull after error: errors:\ndenied: requested access to the resource is denied\nunauthor
Jun 13 12:50:00 p1 dockerd[1477]: time="2022-06-13T12:50:00.492080609+05:30" level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"

Accoring Docker storage drivers | Docker Documentation the overlay2 storage driver only supports xfs with ftype=1 and ext4. extfs is not considered as ext4, is it?

Does this mean you had the data on the old location, changed the data dir without moving the data from the old location to the new location? This might explain why old data is missing. Though, I guess your problem is that you forcefully configured overlay2 with a backed filesystem that doesn’t support it properly. So even if you move the files without fixing this first, it will high likely be still broken.

You need to format your external hd to ext4 and then move existing data to the new location to make it work.
Though: its generaly not recommendable to have the data-root on a removable medium…

I apologize if I confused you.

I had tried to move my docker root to another location, but it did not work. So I completely deleted docker and all files. With the new installation of docker I will keep the docker root at same location as it is default assigned.

Now I created new images and committed them. While committing, I kept in check how much of my storage is occupied. After committing all changes last week, I shutdown my system and when I restarted yesterday I am not able to see any images with sudo docker images and sudo docker ps -l . However the same amount of storage space is occupied as when I could see the images.

I can say this because I had multiple images on which I worked on and the data was around 28GiB and it would have been 28GiB free if all the images were deleted in some case.

$> sudo docker login 
Authenticating with existing credentials...
WARNING! Your password will be stored unencrypted in /home/ashish/.docker/config.json.
Configure a credential helper to remove this warning. See
https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

Even after a successful login the journalctl reports level=info msg="Ignoring extra error returned from registry: unauthorized: authentication required"

update:
I tried to access my image which I had committed before shutting down my system:

$> sudo docker run -it ashish_toolchain_dep2:v2 bash
Unable to find image 'ashish_toolchain_dep2:v2' locally
docker: Error response from daemon: pull access denied for ashish_toolchain_dep2, repository does not exist or may require 'docker login': denied: requested access to the resource is denied.
See 'docker run --help'.

To more ideas:

  1. Your filesystem is corrupted so even if most of the data is in the docker data root, some files are missing, not readable or have the wrong content so Docker don’t know how those files are related.
  2. You have multiple contexts and somehow it was changed after the reboot so the data is there, but the other context stores data in an other directory. It can happen when you install rootless docker or use a new user namespace. The latter is what we would have seen in your folder structure, so that is not the case.

Recommendations:

  1. Image metadata is in /var/lib/docker/image/overlay2. Check the content of “repositories.json” in that folder.
  2. See what you can find in /var/lib/docker/image/overlay2/imagedb/content/sha256. Here are the image metadata fjson iles
  3. Check the content of /var/lib/docker/image/overlay2/layerdb/sha256/ these are the folders for some layer data.
  4. Try to create a new image and check if it works or the filesystem is not usable at all.

The above steps will not solve anything, but you may notice something, for example that everything is empty. In case of filesystem corruption, you need to repair the filesystem or replace the disk. I don’t want to go into details about filesystem corruptions, but you can search for “fsck”

  1. In case you have multiple contexts, you can find them by running docker context ls and switch to default by running docker context use default

Yes, everything was empty. As a test, I pulled the hello-world image and it worked when I run it. I pulled another image made some changes, committed it and it worked. So it seems the filesystem is not corrupted. I am also using other stuff on my machine, and if the fs was corrupted I could have seen some effects over them as well.

I also checked the context and it was set to default. So nothing on this lead as well.

Anyway I decided to do a fsck on root partition, and this method required to make some changes to /etc/fstab .

I found a statement there that I had made changes in previous installation of docker to bind mount to a dir on my other drive. I deleted that statement, rebooted and all my images were visible with sudo docker images .

I am aware that actions of the past come to haunt you, but this was very scary :sweat_smile:

Thanks for all the help. Cheers.