Huge build cache handling

Docker Engine General
,
1

Last night I found my dev server complaining about lack of system space

It was clear it’s not true : I had about 47 GB before hours
but seeking for the proplem I found 54 GB build cache from new language learning tries
some how this is not downloaded via work, it’s just repeated build cache
platform : linux

The question ( disintegrated into parts ) :

  • is there a way to limit docker cache by time or size ?
  • is there a way to tell docker to keep one last copy of the build cache of every image name ?
  • is there an effective way to use cache while reducing it rather than commands like there every time I build image :
    docker buildx build --compress --no-cache --pull=false -t captive .;
docker system prune -f;
docker system df;
echo; docker images;
echo; docker ps --size;
    ?
2

1. Limit Docker Cache by Time or Size

Docker doesn’t directly support limiting the cache by time or size, but you can achieve similar results with scheduled cleanups.

Scheduled Cleanup with cron jobs: You can set up a cron job to prune unused Docker objects regularly.

2. Keep the Last Copy of the Build Cache for Each Image Name

Docker itself doesn’t provide an out-of-the-box way to keep only the last build cache for each image name. However, you can script this behavior.

3

Any basic example of no.2 or even pseudocode for such thing

4

schedualing is not a proper aprouch as you see in my question there was no time limit between growing about 40+ GB
It’s was actually about 4 hours gab to grow that all Giga Bites

5

We clearly need to differentiate between building and running Docker images.

It seems you are talking about building large images, so the use of Dockerfile.

Where is the large data coming from, are models changed that you pull continuously? Are models changed locally that you embed in an image?

Usually docker build is caching layers, but the size shouldn’t increase too much when you always use the same.

The trick in the Dockerfile is to place the stable parts on top and the changing parts at the bottom - and to not ise no cache.

Example: if you have large stable files, add them first, before some dependency/library install and before your own changing code.

That way the first layers are stable and should always stay the same, are only cached once.