I’m looking for some way to clean up the contents of /var/lib/docker/overlay (or /var/lib/docker/overlay2 with overlay2 - I run both, but on different nodes, both seem to have this issue).
Is there any way to map the contents in there to what owns it?
We have some CI runners that nightly do the equivalent of:
but the contents of /var/lib/docker/overlay grows ever larger.
It’s worth noting that sometimes (I can’t purposefully reproduce this) if I run the above clean up commands, stop docker, remove /var/lib/docker/overlay and restart docker when pulling a base image again something still references it, e.g:
# docker run -it --rm ringo/scientific:6.8 bash
Unable to find image 'ringo/scientific:6.8' locally
6.8: Pulling from ringo/scientific
89afeb2e357b: Already exists
Digest: sha256:cb016e92a510334582303b9904d85a0266b4ecdb176b68ccb331a8afe136daf4
Status: Downloaded newer image for ringo/scientific:6.8
docker: Error response from daemon: lstat /var/lib/docker/overlay/ea96bccc6b502595f1b127e5007a8a79c180173e14aba5b1b15703f38e8b5bd4: no such file or directory.
Is there some cache of pulled images that doesn’t show up in docker images -a?
Thanks I’ll add that to the clean up list and see what happens. It’d still be nice to be able to map stuff from the overlay dir to what “owns” it though.
I have exactly the same problem. A few containers running (gitlab + registry), volumes about 2GB overall, But /var/lib/docker/overaly directory is really huge >37GB and removing old containers, images, etc. doesn’t help.
Also I can’t upgrade my docker because it’s a part of CoreOS 1520.6.0 stable release and it can only by 1.12
Same situation happens on another instance with another set of containers (nginx+php+postgres). I felt like it freed some space one day when instance and docker were accidentally rebooted but I’m not sure if it’s a solution
I have same issue - /var/lib/docker/overlay2 has blown to 45 G.
Tried everything but nothing seems to work.
Only option is to drop docker from our scheme of things.
“docker system prune -a” did seem to recover the stuff /var/lib/docker/overlay*, but I had stopped all containers and was happy to lose all the images I had on the machine - I could pull them down again from the local server. Next time I might try without stopping the (few) containers I have going and see whether it is intelligent enough to recover the space then.
/var/lib/docker/overlay2 still grows endlessly even after running all of above and others … if ANYONE finds a real solution please post it here … I am using latest docker Version: 18.05.0-ce so this docker bug lives on ;-(((
I’m having a similar issue. I’m trying to reset a mysql database to its initial state but no amount of volume or container or image removal is able to erase the data I’ve put in the db. I’ve followed all of the standard paths and have traced the cached data back to the overlay2 directory. There needs to be a good way to clean this up. I’ll be opening up another issue about this.
Well, first of all, I would like to say that this kind of bug is the reason many people complain and say bad thing about open source / free software. This kind of problem that sticks forever with no solution most of times brings infeasibility of use.
Anyway, I am facing this problem and I am proceeding with some investigation. I discover some things about this issue. In my case, I have a dockerfile with Wildfly image that copies a .war application and rebuilds it every deploy and the image is reloaded into docker and started again after removing the old container.
I tried the above clean up command but nothing works. In the docker inspect command I found that section:
So, in the overlay folder you many folders with content of different containers. It seems docker keeps track of all runs of the same image:version plus the container name. Everytime I redeploy my image is with the same container name, image name and tagged version (latest), so it keeps track of the diff folder of the previous runs.
When I found the container owning the 40GB folder inside overlay, I stopped it and removed it with docker stop and docker rm and the folder was cleaned up to 0 bytes. I redeployed everything and the initial size was small. I discovered that it was a problem with my app that grows the log too much, but removing the container and redeploying solved the problem.
I suspect this problem is related with problems of the images, not necessarily a docker problem. Anyway, there is no easy path to find and clean this up. Let’s do some more investigation.
I was able to clean it up while upgrading the Docker version.
Stopped Docker (sudo systemctl stop docker)
Uninstalled Docker (sudo yum remove docker-ce)
Nuked the directory (sudo rm -r /var/lib/docker/overlay2),
then re-created the directory (sudo mkdir /var/lib/docker/overlay2).
Then re-installed Docker (sudo yum install docker-ce)
And restarted the Daemon (sudo systemctl start docker)
You need to shut down everything before doing this, and you may lose containers/volumes/etc. I have no idea if you can just do this without uninstalling and re-installing.