Docker overlay folder gets filled up fast

General
, ,
1

We have currently 9 static agents on our Jenkins controller, and all are identical along with labels as “docker”.

All the linux agents are installed with docker has capacity of 16 CPU Cores | 640 GB Storage|32 GB RAM |1 Volume

Problem: /var/lib/docker/overlay folder gets filled up every single day disturbing our builds and also conflicting networks. We also have cron setup to do regular cleanup using docker system prune but that is also not helping much.

We also tried to leverage mapping of var/lib/docker to external volume that also is getting filled up very fast. Seeking your advise to fix the issue.

Note : Even though we perform docker system prune we see lot of older files like tar files,image layers,build cache are there in overlay folder which are 2 years old. In this case system prune is not working.

Should we need to execute : /usr/bin/docker system prune -

Is there any specific difference between docker system prune & /usr/bin/docker system prune -

2

Looks like you don’t use dynamic docker agents, and instead use static agents that retain state. It is best practice to use untainted build containers, or in case of Jenkins a dynamic docker agent.

I just googled “jenkins dynamic docker agents”, and this is the first result.

I am sure there are plenty of more blog posts to help to set it up properly.

Though, generally I would recommend investing the time into planing and performing a migration to a more modern CI/CD environment. Keeping a Jenkins instance patched and secure burns a lot of work hours, especially when something goes wrong, or another plugin becomes unmaintained, or another critical security finding requires immediate patching.

I can not answer any follow-up questions, as it has been years since I had to use Jenkins.