Docker Community Forums

Share and learn in the Docker community.

Some way to clean up / identify contents of /var/lib/docker/overlay


(Lyte) #4

So ```docker system prune -a -f` doesn’t help.

I’ve upgraded docker (and rm -rf /var/lib/docker/* during the upgrade, so this isn’t from before the upgrade):

# docker info | egrep '^(Server Version|Storage)'
Server Version: 17.03.1-ce
Storage Driver: overlay2

Docker isn’t aware of any containers or images:

[root@*** ~]# docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES
[root@*** ~]# docker images -a
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE

Neither prune command helps:

[root@*** ~]# docker volume prune -f
Total reclaimed space: 0 B
[root@*** ~]# docker system prune -a -f
Total reclaimed space: 0 B

There’s a lot of dirs left behind in /var/lib/docker/overlay2:

[root@*** ~]# ls -la /var/lib/docker/overlay2/ | wc -l
113

Taking up a fair bit of space (given they don’t seem to “belong to anything”):

[root@*** ~]# du -sch /var/lib/docker/overlay2/*
40K	/var/lib/docker/overlay2/0507f5e8d32952a9ccf45bad488c084f017d9db3e153d6f22e46532edadef58e
80K	/var/lib/docker/overlay2/0cc693976ed3d560f0a13c03ce5ab60645e332da1d97b9c775ebc9d7dde233ae
36K	/var/lib/docker/overlay2/0db10721d06e14dc934f9a54313885e3a5a7e53cc0473897e53a0f9ec68d0ee6
169M	/var/lib/docker/overlay2/0ff14195d187ce0effa72423b380fff6c4b5a1efbcdca9e8ba5eff8e99147fd8
[...]
321M	/var/lib/docker/overlay2/cb45cb0ac04e59f328f05379aa76126eaa6d9bbd10e0956861f0b8a2ca9f505b
119M	/var/lib/docker/overlay2/cf0228e906c84f5ba1ec918a5a5872f077fbf572870cdb414899188d29ea807a
586M	/var/lib/docker/overlay2/d4490f059916dcf64e3cd95a0af310865b293bcc1e89d4ab68051744e383b5aa
[...]
5.7G	total

There’s gotta be some way I can map from the contents of that directory back to what owns it?


(Lyte) #5

I’ve posted a bug report at https://github.com/docker/docker/issues/32420 as based on https://github.com/docker/docker/issues/21925 this is considered a bug there’s just been no repro cases previously (hopefully I can get a shareable one…).


(Think) #6

just to be curious, could you also execute a docker volume ls?


(Lyte) #7

Comes up empty (not using volumes).


(Evertramos) #8

Just to keep the record, I have done some clean up using:

docker volume rm $(docker volume ls -qf dangling=true)

This will not delete any container or any volume in use!


(Vladkras) #9

I have exactly the same problem. A few containers running (gitlab + registry), volumes about 2GB overall, But /var/lib/docker/overaly directory is really huge >37GB and removing old containers, images, etc. doesn’t help.

Also I can’t upgrade my docker because it’s a part of CoreOS 1520.6.0 stable release and it can only by 1.12

Same situation happens on another instance with another set of containers (nginx+php+postgres). I felt like it freed some space one day when instance and docker were accidentally rebooted but I’m not sure if it’s a solution


(Sureshskit) #10

any update on this? we are also facing the space issue.


(Vladkras) #11

No. We decided to buy more disk space, because official page explicitely says:

At this time, we have traded off simplicity and ease of deployment for disk space.


(Geekifier) #12

Thanks, this was exactly what I needed :).


(Pingal) #13

I have same issue - /var/lib/docker/overlay2 has blown to 45 G.
Tried everything but nothing seems to work.
Only option is to drop docker from our scheme of things.


(Johnfo) #14

“docker system prune -a” did seem to recover the stuff /var/lib/docker/overlay*, but I had stopped all containers and was happy to lose all the images I had on the machine - I could pull them down again from the local server. Next time I might try without stopping the (few) containers I have going and see whether it is intelligent enough to recover the space then.


(Eldeberde) #15

Hi. I’m using this image to clean old volumes and images every day automatically:

https://hub.docker.com/r/clockworksoul/docker-gc-cron/


(Scott Stensland) #16

/var/lib/docker/overlay2 still grows endlessly even after running all of above and others … if ANYONE finds a real solution please post it here … I am using latest docker Version: 18.05.0-ce so this docker bug lives on ;-(((


(Dustinschultz) #17

I’m having a similar issue. I’m trying to reset a mysql database to its initial state but no amount of volume or container or image removal is able to erase the data I’ve put in the db. I’ve followed all of the standard paths and have traced the cached data back to the overlay2 directory. There needs to be a good way to clean this up. I’ll be opening up another issue about this.


(LuPa) #18

Same here. Any solution ?


(Renatorro) #19

Hi there.

Well, first of all, I would like to say that this kind of bug is the reason many people complain and say bad thing about open source / free software. This kind of problem that sticks forever with no solution most of times brings infeasibility of use.

Anyway, I am facing this problem and I am proceeding with some investigation. I discover some things about this issue. In my case, I have a dockerfile with Wildfly image that copies a .war application and rebuilds it every deploy and the image is reloaded into docker and started again after removing the old container.

I tried the above clean up command but nothing works. In the docker inspect command I found that section:

"GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e-init/diff:/var/lib/docker/overlay2/4b6ccf942c76c0ea2c228afaff989f000847be1fed361ed2dabc38f15df2fe1f/diff:/var/lib/docker/overlay2/c309a80b5965c0b65308cd766fc1a996da08f96bf920c4b7ff69f05337f61535/diff:/var/lib/docker/overlay2/005195bc5bfb582da94a6054a2f0fdb1e214075916a2d759a42c801ad2175897/diff:/var/lib/docker/overlay2/1e8844c07e7e3b2ae6504bd85d4e3cfa2bcdfb73b9c54107e56404ecb66c784f/diff:/var/lib/docker/overlay2/0ade4795c69216431991716ea03448a9b79a2e2a72c03513a87b3b8e30f7fe79/diff:/var/lib/docker/overlay2/866e1f845c6ba69501b16e24441e74e80291f03b29446d10977fcd4f27afd1fc/diff:/var/lib/docker/overlay2/438f88a38ba85e10106e971ba966085b37770dfc547dc4b43fe424bf1aad4480/diff",
                "MergedDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/merged",
                "UpperDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/diff",
                "WorkDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/work"
            },
            "Name": "overlay2"
        },

So, in the overlay folder you many folders with content of different containers. It seems docker keeps track of all runs of the same image:version plus the container name. Everytime I redeploy my image is with the same container name, image name and tagged version (latest), so it keeps track of the diff folder of the previous runs.

When I found the container owning the 40GB folder inside overlay, I stopped it and removed it with docker stop and docker rm and the folder was cleaned up to 0 bytes. I redeployed everything and the initial size was small. I discovered that it was a problem with my app that grows the log too much, but removing the container and redeploying solved the problem.

I suspect this problem is related with problems of the images, not necessarily a docker problem. Anyway, there is no easy path to find and clean this up. Let’s do some more investigation.


(Aostreetart) #20

I was able to clean it up while upgrading the Docker version.

Stopped Docker (sudo systemctl stop docker)
Uninstalled Docker (sudo yum remove docker-ce)
Nuked the directory (sudo rm -r /var/lib/docker/overlay2),
then re-created the directory (sudo mkdir /var/lib/docker/overlay2).

Then re-installed Docker (sudo yum install docker-ce)
And restarted the Daemon (sudo systemctl start docker)

You need to shut down everything before doing this, and you may lose containers/volumes/etc. I have no idea if you can just do this without uninstalling and re-installing.


(Dpatekar) #21

I was wondering the same thing some time ago.
It’s not a bug, it’s a feature :slight_smile:

du -sh /var/lib/docker/overlay2
is not showing objective value because merge folders have been mounted using overlay driver and du output is not actual disk allocation size.

You can see the actual disk allocation size by examining only diff folders like:
du -shc /var/lib/docker/overlay2/*/diff

You can test this in your environment like this:
run
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2
Now start 20 centos containers and observe what has change:
for i in {1..20}; do docker run -itd centos bash; done
df -h /dev/sd*
du -shc /var/lib/docker/overlay2/*/diff
du -sh /var/lib/docker/overlay2

You can see that the actual disk allocation (df command) is just cca 200MB more than before, but “du” on whole folder outputs 4.2G allocation.
“du” on “diff” folders shows 212M what is correct.

This is how Docker works and what makes it great!


(Oscaralmgren) #22

So we had the same issue (using Gitlab + private registry) and found this solution to work.

sudo gitlab-ctl registry-garbage-collect

This command will remove images in the directory
/var/opt/gitlab/gitlab-rails/shared/registry/docker/registry/v2/blobs

We reclaimed our disk space usage from 89% to 17%.

Note: we also remove old unused images (tags) from the Gitlab web ui under project settings > registry > list containers in repo.

Edit: more info to be found here https://github.com/sciapp/gitlab-registry-cleanup


(Vladkras) #23

Good to hear, but this topic is about docker itself, not about registry - it’s a separate application