Huge disk usage

Docker Engine General
1

Can someone explain my docker disk usage?
Is this normal?

I have a docker host running in a LXC that use way to mouch space i think. I have had to increse drive multiple times. Due to backup size and host disk usage i would like to not having to over provission if i can reduce this disk use.

LXC is running Ubuntu 24.04.3 LTS x86_64

All docker images combined in portainer is 18,276Gb (No unused images)
All data mapped to docker hosts are stored in one folder that is 2,8GB
I do not have any volumes in docker, so 0mb

So total disk usage should be 18,276Gb (images) + 2,8Gb (data mapped to volumes) = 21.076Gb

LXC container disk usage is:
/var/lib/containerd/ 27,1GB
/var/lib/docker/ 9,3 Gb
/home/username/docker/ 2,8 Gb

Actual total disk usage is 27,1Gb (containerd folder) + 9,3Gb (docker folder) + 2,8Gb (data mapped to volumes) = 39,2Gb

Is portainer showing wrong image size?
The total mismatch is 18.124Gb

2

Why run Docker in LXC in the first place? Seems to me like double isolation and double trouble with assigning CPU, RAM and disk space.

Disk space is usually taken up by Docker images, outdated Docker images, Docker containers and stopped Docker containers. First step would be to remove unused/old images and unused/old containers including volumes.

Then make sure you don’t create too much console output within the containers and don’t create (temporary) files within the container.

3

None of that realy answerd the question.
But i’ll give a short explanation anyway.

  • lxc running on proxmox, so actualy more effective than using a vm. not relevant for the question realy, but it is why i do not want do keep incresing the disk image if i do not have to. Explained in original post
  • All unused images removed. See original post
  • I have no volumes. See original post
  • Logs are located in /var/lib/docker/containers//-json.log and we are only talking megabytes her. no issue.
  • Temporary files i do not know about. i usualy map directorys that the container use to a local path. so i thought i had control of this.

And when i run docker image prune -a i get this:

WARNING! This will remove all images without at least one container associated to them.
Are you sure you want to continue? [y/N] y
Total reclaimed space: 0B
4

The way you collect the folder sizes is most likely wrong, see https://forums.docker.com/t/huge-increase-in-lxc-backup/150886/3.

Furthermore, I can not tell whether Portainer actually shows the real image size. I don’t use Portainer, I feel it’s cumbersome to use, and I am not patient enough for “clickOps”. The containerd image store holds a compressed and an uncompressed copy of the image, not sure if Portainer “knows” this.

Note: this is true for fresh Docker Engine v29+ installations. Existing installations that update to v29+ don’t use the containerd image store, unless you configure it specifically use it! Since most of your data is located in /var/lib/containerd you must be using the containerd image store.

5

Yes. thanks for that. And i agree with the clickOps. i only use Portainer for an overview. and the occasional restart/remove
i did a major change to running everything on proxmox, and got the new storage driver at the same time on all my LXC’s and VM’s using Docker

If i understand things correctly
Docker Root Dir /var/lib/docker/rootfs/overlayfs contains folders for the running image. if i stop a container, the folder for it disappears, but not much changes in /var/lib/containerd

and /var/lib/containerd contains images and uncompressed layers.

So disk usage reported by using ncdu is real. portainer is only reporting single downloaded image size. not all the uncompressed files and layers that containerd has control over.

But “take something with a pinch of salt”

I have removed some large images that i did not relay use on this host.
That saved a lot of space in both folders.
One simple service had an image that was over 5gb. Ruthless of the container maker. (looking at you stirlingPDF)

But i think i also have an orphan dir in /var/lib/containerd

is there any way of figuring out what container has winch folded in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots?
it contains numbered folders like: 761 774 783 852 861 870 893 902 911 920 93 94 989
1000 101 1019 1036 1067 1083 109

and 771 is the largest by far at 2Gb, but what container is that? how to find?
Small update just after posting this: 771 does not show up when i run mount and if it is orphaned i can not find the container it belongs to. so i guess i need to filter folders to see what container they belong to and see if i find one that does not belong to a container.

I’m considering removing all containers and rebuilding. i already have a script for it, and all data should be safe. I also have a backup of the vm… but still. i have to do it when i have the time to fix it if it breaks :stuck_out_tongue:

This is the answer googles ai gave me. winch is a good summary of it:

Containerd typically reports higher real disk usage than legacy drivers because it stores both compressed and uncompressed image layers, often causing disk usage to double. Images, containers, volumes, and build cache consume space, which can be monitored via docker system df or by checking /var/lib/containerd/ or /var/lib/rancher/k3s/agent/containerd/.

Key Reasons for High Disk Usage

  • Dual Storage: Containerd stores images in both compressed (received) and uncompressed (unpacked) formats to allow for faster image pulling and pushing.
  • Snapshotter Storage: While uncompressed layers are de-duplicated, the extra space used by keeping compressed blobs adds overhead.
  • Redundant Images: In some scenarios (e.g., K3s/Kubernetes), identical images might be stored multiple times, wasting significant space.
  • Build Cache: Over time, unused build caches can accumulate, taking up substantial storage.

How to Monitor and Reduce Real Disk Usage

  • Check Usage: Use docker system df -v to get a detailed view of space used by images, containers, and volumes.
  • Prune Data: Regularly run docker system prune -a --volumes to remove unused images, containers, and volumes.
  • Check du: Run du -sh /var/lib/containerd or the relevant data directory to see the actual space consumed on disk.
  • Clean Up Volumes: Volumes are not automatically removed by prune, so ensure unused volumes are deleted.
  • Restart/Reset: For extreme cases in development, restarting the service or performing a prune can free up space.
6

For /var/lib/docker: True, if ncdu -x <path> was used. Not true, if -x was not used, as files are counted multiple times from the overlay filesystem.

Same reason as above: without -x things are counted multiple times.

My first though: that’s a crime!
My second thought: there are use cases where huge image sizes are valid, e.g. if you have a Swiss army knife tool image used in cicd pipeline jobs, or have huge dependency list or embedd a “small” ai model in the image.

No idea.

7

Thanks for the quick reply :slight_smile:
That was quite the difference on the docker folder, but not on containerd folder
i guess what is left in the docker folder that counts as used space, are junk files that the container writes outside of mounts. is that a correct assumption? i see it mostly json.log files that take up space.

ncdu -x
was not able to attach image. folders below same but docker folder only 343mb

ncdu (without -x)

image
image501×299 17.2 KB

I use pulse for overview and alerting on proxmox and docker.
So it have to check if that reports real disk usage or not.

Does df -h report real disk usage? that is what i mostly use.
i’m glad i did not just keep increasing the lxc disk due to this. i’m down at 66% now, pulse reports 62%.

But it’s ok with some growing room :stuck_out_tongue:

i guess i have to do a wipe and rebuild at some point. go get rid of orphans if i actually have any

8

Each container file system consists of the read only image layers and a copy-on-write write layer. Whatever files are changed or added, and are not located in a folder mapped out using a volume, end up in the write layer, which should be physically located underneath the rootfs folder. Everything you see in docker diff <container name or id> should be physically written into write layer of the container’s rootfs folder.

In the containerd folder, the compressed image layers are in io.containerd.content.v1.content, the extracted in io.containerd.snapshotter.v1.overlayfs

It does.

9

Thanks for clearing this up.

Hope it is helpful for others also
:slight_smile:

10

By the way.
Deleted the 771 folder and everything still works. Even after restart.

There should be a consistent way to find orphaned folders in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

11

Apparently this isn’t true with the containerd snapshotter anymore. the folder io.containerd.snapshotter.v1.overlayfs does not only have the extracted image layer folders, but also has the final mutable folder used as container filesystem. I guess the rootfs folder only mounts the containerd represenation of the conainer filesystem.

Please do not delete any folders you deem irrelevant, this might break the chain of extracted image layers required for a container file system.

You can try if ctr --namespace moby <ctr arguments> helps to detect it.

Worst case, you should be able to use a tool like bbolt (requires extra installation) to query data from the bolt database file metadata.db.

12

I know i should not just delete folders, but i took the chance. :slight_smile:
Also, i saw all the other folder id’s when i ran mount but not the one i deleted

I tried sudo ctr --namespace moby containers list and that gave me a list of container ids that match with folders in /var/lib/docker/containers

But i’m not able to find anything that shows the folders in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots

I prefer to not have to install all kinds of software and runtimes on my docker lxc. So waiting a bit with bbolt.
I guess i can copy the file out of the lxc and mess with it if i need in the future.
But there should be a command to see the connection somewhere. and it looks like there is a pull request for it, see links

ctr:add mountInfo command to print all snapshots mount info by ningmingxiao · Pull Request #11760 · containerd/containerd · GitHub (stale due to no activity?)

13

The Issue you found is spot-on! It’s a pity it’s open since ~ 18 months..

Thanks to our discussion I noticed that ctr -n moby content ls showed a inconsistency compared to docker image ls. Even when the latter showed no image, the first still showed image layers with labels of images that were deleted in docker long time ago.

This might be a bug from the early docker v29 versions. Not sure if it’s fixed or not.

14

Looks like others have had similar garbage collection issues.
There are multiple forum posts and i also found this guide for microk8s to reset the overlay file system. Cleaning the containerd OverlayFS Directory in MicroK8s :: Martin Koníček - Personal site

i’m considering making a script that can compare folders in /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots with the output of mount -t overlay
but that is for another day.

And i do not like deleting folders manually in there.I made a comment, linking to this thread on the pull request. We will see if that makes a difference

15

I made this: Find orphaned snapshots
And it indicates i have more folders that are not in use, but i might be wrong

Can someone else test it?

image
image694×1012 75.3 KB