Some way to clean up / identify contents of /var/lib/docker/overlay

[dataheadless]

@lyte

I’m looking for some way to clean up the contents of /var/lib/docker/overlay (or /var/lib/docker/overlay2 with overlay2 - I run both, but on different nodes, both seem to have this issue).

Is there any way to map the contents in there to what owns it?

We have some CI runners that nightly do the equivalent of:

docker rm -vf $(docker ps -aq)
docker rmi -f $(docker images -aq)
docker volume prune -f

but the contents of /var/lib/docker/overlay grows ever larger.

It’s worth noting that sometimes (I can’t purposefully reproduce this) if I run the above clean up commands, stop docker, remove /var/lib/docker/overlay and restart docker when pulling a base image again something still references it, e.g:

# docker run -it --rm ringo/scientific:6.8 bash
Unable to find image 'ringo/scientific:6.8' locally
6.8: Pulling from ringo/scientific
89afeb2e357b: Already exists
Digest: sha256:cb016e92a510334582303b9904d85a0266b4ecdb176b68ccb331a8afe136daf4
Status: Downloaded newer image for ringo/scientific:6.8
docker: Error response from daemon: lstat /var/lib/docker/overlay/ea96bccc6b502595f1b127e5007a8a79c180173e14aba5b1b15703f38e8b5bd4: no such file or directory.

Is there some cache of pulled images that doesn’t show up in docker images -a?

If anyone still having issues, this article is helpful Optimizing Docker Storage