And up to now that has made sure that all my images and stuff has been saved to an USB3 SSD on all my Pi 4 and Pi 5’s. But when I did an update on one of them today (the one I always do updates on first because it’s not essential/production), the download filled the SD card instead of going to the SSD. But not to /var/lib/Docker which doesn’t even excist.
docker info shows Docker Root Dir: /media/pi/Docker/Docker-System and the directory contains about 6 gigs of files, which is correct. But still the system and Docker now uses more than 15 gigs on the SD card, while it only filled around 8 before this started. It was when pulling that the card space was used up. I have searched and looked, and I can’t understand where all that extra space is used.
I have not dared to test with the rest of the pack if it’s a one off thing, but I have gone back to an earlier image of the same Pi and updated again, and it gives the same result. So is this a new behavior that is to be expected, and I need to do something else to get the files in another place, or is this something wonky in this Pi or my setup?
This explains the behaviour: the installation is using the containerd snapshoter as image store. This is the default behaviour for fresh installations starting with docker v29. As a result the majority of data is not written into the docker-data folder, but instead delegated to containerd, and stored somewhere under /var/lib/containerd.
We had a couple of topics about this in the forum, the search should find them. If I remember right, at least in one of those topics, the setting to configure the containerd data folder can be found as well.
Thanks, you’re correct! That’s the problem, but that’s not the explanation. I saw that stuff in the manual, but didn’t think it was relevant to me because as I said this is not a new install. It an installation from at least two years back, when I went Bookworm on the Pi. And it has been updated since then, but not with a new installation.
This happened a week or so ago, and I thought I had done something wrong, so I went back to a backup from February. When updating that (still no new installation of Docker) it had the same thing happening, so I understood that it had to be something with the update.
Tomorrow I will now go back to the same backup from February, add the config file and then try another update. I assume that will go without those problems. So yet another 4-5 hours of my life (with searching the web and forum without finding the info I thought was relevant) lost to 's most diabolical invetion so far: updates!
Btw another thing that happened the first time I did this, but not now, was that a bunch of “Merged” and a series of seemingly random letters and numbers appeared in places. That has not happened this time.
Make sure you use a dedicated directory for each daemon. If two daemons share the same directory, for example an NFS share, you will experience errors that are difficult to troubleshoot.
Does that mean that using /media/pi/Docker/Docker-System for Docker and /media/pi/Docker/Containerd-System for Containerd will be a problem, or will that work?
Using the containerd image snapshotter does explain the behaviour. Though, it does not why it was selected on an existing installation that is updated.
Since there is no mention in v29 release notes up to 29.6.1, It looks like a bug. If the storage driver would be forced on updates, I would expect a mention in the release notes.
Short-Term you could revert back to the overlay2 storage driver: OverlayFS storage driver | Docker Docs. Still, you might want to migrate to the new storage backend on your timeline before it gets deprecated and forces you to migrate right away.
Since both paths are different, you already use dedicated directories.
Thank you! I have added the storage for containerd and it seems to be working again. Oh, and I meant that it didn’t explain why it happened when I wasn’t doing a fresh install, but I guess the bug theory is probably sound.
I need to add one thing, mainly for my own sake, so I can use this for reference: Either the documentation is wrong, or something’s weird with all my systems, both Bookworm and Trixie. Because using this is what’s in the documentation:
version = 2 root = "/mnt/containerd-data"
version = 2 over the root for containerd, messes up every system. Without it everything works well.
Its most likely either is a bug, or the containerd version is < 2.0. The version string from the docker info output shows a checksum or id, instead of the version number. You might want to check it with dpkg -l | grep containerd.
Update: I made an assumption which might not be true: I assumed that config version = 2 is used in containerd >=2.0 , without checking the release notes of containerd.. I have no idea in which containerd version this config version was introduced.
update2: I just looked it up:
Version 3 (Recommended for containerd 2.x): Introduced in containerd 2.0. Several plugin IDs have changed in this version.
Version 2 (Recommended for containerd 1.x): Introduced in containerd 1.3. Still supported in containerd v2.x. Plugin IDs are changed to have prefixes like “io.containerd.”.
Version 1: Introduced in containerd 1.0. Removed in containerd 2.0.
's most diabolical invetion so far: updates!