A server that runs a bunch containers gradually runs low on disk space. Most of the usage seems to be under /var/lib/docker/overlay2 even though the containers dont’ store much data. Some space frees if clean up images or stop containers but it fills back later on. Now what to do? Can I store data on cloud storages like Google drive, box or any? Or just what to do?
The overlay2 folder stores all the overlay2 filesystem. It includes image layers, container filesystem and build caches.
You can use docker system df to get more info about sizes of docker objects and use docker systen prune or othe rprune commands to delete data, including the build cache.
Pruning the build cache is not menitoned on the above documentation, but you can use
docker builder prune
Don’t forget to check the help of all commands for their parameters.
The size of that overlay2 folder can also increase if you build images incorrectly. For example there were cases when users thougght they could change an existing image and they used the same image tagin the FROM instruction of a Dockerfile as they used to name their built image. That way they just added new layers to the original image and even if they thought they deleted files, they never did as each layer can only hide files in previous layers.
Makes sense Thanks. But how do you tell if a dockerfile’s layering stuff wrong and making overlay2 blow?
I would start by checking the containers.
If it’s just plain containers running, taking up more space over time, then I would assume they write output to docker log or inside container.
Find the large container, check with docker logs for extensive log usage, this can be limited (doc). Otherwise go into container and check folder size inside, from root going deeper, if large files are written into container during runtime.
Thanks! Is there a way to make docker auto limit or rotate logs so they don’t pile up like that?
I already linked the doc to limit Docker logs. If the applications writes into its own log file inside the container, you need to create your own solution, that’s not Docker responsibility.
Oh, got it. Thank you Blue!