Docker not loading containers from filesystem after reboot

General
1

Hey there,

I have a problem with my docker environment since I recently restarted my system.

I have the following setup:
Installed Version: Docker version 23.0.1, build a5ee5b1
OS: Ubuntu 22.04.2 LTS running as LXC container in Proxmox

After an update of Proxmox and a restart of my host and the dockerhost-LXC, docker is not starting/listing my previous containers anymore:

docker container ls -a
CONTAINER ID   IMAGE     COMMAND   CREATED   STATUS    PORTS     NAMES

docker images -a
REPOSITORY   TAG       IMAGE ID   CREATED   SIZE

‘docker volume ls’ does list my volumes however.

Looking into the filesystem in /var/lib/docker I see the folders of my containers in /var/lib/containers. For some reason the docker daemon does not find them.
The structure in each folder looks like this:

root@dockerhost:/var/lib/docker/containers/89df575bae2dc97444893372522ed293d13089df5afb9168ebfed25f7424ffd2# tree .
.
|-- 89df575bae2dc97444893372522ed293d13089df5afb9168ebfed25f7424ffd2-json.log
|-- checkpoints
|-- config.v2.json
|-- hostconfig.json
|-- hostname
|-- hosts
|-- mounts
`-- resolv.conf

Does someone have any idea how I could fix that?

Thanks in advance!

2

I would put my money on “the determined default storage driver changed”.

Containers and images are bound to a storage driver when pulled/created. If the storage driver changes, it will look like existing images and containers are gone. Though, what really happens is that the changed storage driver isn’t able to work with existing images and containers from the previous storage driver.

You can check the storage driver the container uses like this:

jq -r '.Driver' < /var/lib/docker/containers/89df575bae2dc97444893372522ed293d13089df5afb9168ebfed25f7424ffd2/config.v2.json

Furthermore, you want to check the currently used storage driver:

docker info --format '{{.Driver}}'

Please share the output of both commands, plus the output of cat /etc/docker/deamon.json. Chances are high, the file does not even exist. All commands should be run as root.

With this three information, we can recommend how to approach further. Just setting the storage-driver to the old one might look like a quick win, but it might not be the best option.

2 Likes
3

Seems like you had the right guess here, though I wonder how that happened.
The config.v2.json says vfs, while ‘docker info’ returns ‘overlay2’

jq -r '.Driver' < /var/lib/docker/containers/89df575bae2dc97444893372522ed293d13089df5afb9168ebfed25f7424ffd2/config.v2.json
vfs

docker info --format '{{.Driver}}'
overlay2

/etc/docker/deamon.json does not exist

4

My bad, it should have been /etc/docker/daemon.json. If it doesn’t exist, you can create it.

Seems the overlay2 support wasn’t there earlier. It is definitely the storage driver you want to go with, as it’s the most performant and efficient storage driver. Vfs on the other hand is everything but efficient (=wast’s a lot of space), is slower and according the docs not recommended for productive systems.

Personally, I would switch back to vfs to clean all images and containers, and start over with the overlay2 storage driver. Neither images nor containers are worth being backed up, as you can pull or build an image again, and can easily create containers again. Volumes on the other hand would be wort to be backed up, but they are not affected by teh storage driver → you can use them as they are


You can switch back the storage driver according this Use the VFS storage driver | Docker Docs. Once you made your cleanup, I highly recommend switching to overlay2 permanently.