How can I cleanup OVERLAY2 folder

Hey Gents

running ard 7-10 containers on my Debian 11,I found my /var/lib/docker/overlay2 folder increase abt 300MB daily and it is endless

I did some homework here and known that is folder is something about docker file system and tried all prune comments with varity of options ,but can not reclaim any space.Since I dont know how to delete something yet ,I should at least firgure out what is growning
so I run flwg commends every morning

df -h
cd /var/lib/docker/overlay2
du -h --max-depth=1

df -h shows /var/lib/docker/overlay2 grows about 300MB daily
but the after checking du comment output ,I found the outputs are exactly same as yestaday

df reports growing but du reports same

I learned that du/df has different calculation ,but how can I fix this problem?

it is fine that overlay take a lot of space to me but I cant leave it growing endlessly without deploy new containers

docker -version
Docker version 20.10.13, build a224086

Try:

du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker/overlay2

du ist not realy equiped to determine the size for a “composed” filesystem like overlay2, where you have a lower, upper, merged and workdir, thus counting the very same files more than once.

df on the other hand shows the real occupied size.

You might want to regularly check du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker and see where the growth actualy happens.

If you would have said that df shows growth, it might have been explainable with logs beeing written in the container filesystem. If the growth is not in the overlay2 it could still be container logging gone wild.

Thanks Meyay

yes, I also learned du output is not real occupation ,which I am fine with that

You might want to regularly check du --human-readable --max-depth 1 --no-dereference --one-file-system /var/lib/docker and see where the growth actualy happens.
yes, I only focused on the /var/lib/docker/overlay2 but overlooked the docker folder and I will check in the future

If you would have said that df shows growth, it might have been explainable with logs beeing written in the container filesystem.
I shutdown couple of containers ,remain one or two really essential container running and I can found by df that growth is still there, about 1MB per hour ,more or less, longer or shorter

may I know if that would be a good idea to locate where the growth is by du /var/lib/docker?

thanks

B.RGDS

Hello Meyay

further below

df on the other hand shows the real occupied size.
but when I df -h ,it shows overlay takes great space in my system ,and when it eats all space, the dokcer would return error

how can I deal with that too

here are my df -h output

文件系统 容量space 已用used 可用avail 已用%used% mount point

/dev/mapper/Debian–vg-var 8.1G 7.5G 157M 98% /var

/dev/mapper/Debian–vg-home 92G 1.6G 86G 2% /home
/dev/mapper/bookVG-bookLV 459G 244G 192G 56% /home/bing/NAS/book
/dev/mapper/mailVG-mailLV 459G 602M 435G 1% /home/bing/mail
/dev/mapper/cloudVG-cloudLV 1.4T 669G 647G 51% /home/bing/super_nextcloud/app/data
/dev/mapper/videoVG-videoLV 12T 8.7T 2.6T 78% /home/bing/NAS/video
/dev/mapper/softwareVG-softwareLV 2.3T 884G 1.3T 41% /home/bing/NAS/software
/dev/mapper/pictureVG-pictureLV 917G 443G 432G 51% /home/bing/NAS/picture
tmpfs 796M 0 796M 0% /run/user/1000

Well, df shows how the free/used capacity of a partition is, not how much a folder in the partition uses.
Generaly you will have a hard time to run containers with just 8.1G for /var shared with the system.

You will want to have at least 30G, better 40G in a dedicated volume group mounted into /var/lib/docker.

30 or 40 even larger is fine ,that is an LVM device in my system ,easily to expand

but the key point is ,I need to locate where and why the size is growing

This is a way to figure out if it’s realy the overlay2 or maybee the container subfolder.

Make sure the containers ment to be run permanently are not logging with a log-level of debug or trace.

Thanks Meyay

trying this