Some way to clean up / identify contents of /var/lib/docker/overlay

General
19

Hi there.

Well, first of all, I would like to say that this kind of bug is the reason many people complain and say bad thing about open source / free software. This kind of problem that sticks forever with no solution most of times brings infeasibility of use.

Anyway, I am facing this problem and I am proceeding with some investigation. I discover some things about this issue. In my case, I have a dockerfile with Wildfly image that copies a .war application and rebuilds it every deploy and the image is reloaded into docker and started again after removing the old container.

I tried the above clean up command but nothing works. In the docker inspect command I found that section:

"GraphDriver": {
            "Data": {
                "LowerDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e-init/diff:/var/lib/docker/overlay2/4b6ccf942c76c0ea2c228afaff989f000847be1fed361ed2dabc38f15df2fe1f/diff:/var/lib/docker/overlay2/c309a80b5965c0b65308cd766fc1a996da08f96bf920c4b7ff69f05337f61535/diff:/var/lib/docker/overlay2/005195bc5bfb582da94a6054a2f0fdb1e214075916a2d759a42c801ad2175897/diff:/var/lib/docker/overlay2/1e8844c07e7e3b2ae6504bd85d4e3cfa2bcdfb73b9c54107e56404ecb66c784f/diff:/var/lib/docker/overlay2/0ade4795c69216431991716ea03448a9b79a2e2a72c03513a87b3b8e30f7fe79/diff:/var/lib/docker/overlay2/866e1f845c6ba69501b16e24441e74e80291f03b29446d10977fcd4f27afd1fc/diff:/var/lib/docker/overlay2/438f88a38ba85e10106e971ba966085b37770dfc547dc4b43fe424bf1aad4480/diff",
                "MergedDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/merged",
                "UpperDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/diff",
                "WorkDir": "/var/lib/docker/overlay2/fa8c221a6be224f62cc89b01ed2e3332d039a973315497a1fd08eeadbfd8e89e/work"
            },
            "Name": "overlay2"
        },

So, in the overlay folder you many folders with content of different containers. It seems docker keeps track of all runs of the same image:version plus the container name. Everytime I redeploy my image is with the same container name, image name and tagged version (latest), so it keeps track of the diff folder of the previous runs.

When I found the container owning the 40GB folder inside overlay, I stopped it and removed it with docker stop and docker rm and the folder was cleaned up to 0 bytes. I redeployed everything and the initial size was small. I discovered that it was a problem with my app that grows the log too much, but removing the container and redeploying solved the problem.

I suspect this problem is related with problems of the images, not necessarily a docker problem. Anyway, there is no easy path to find and clean this up. Let’s do some more investigation.

4 Likes