Docker files fills up disk space

Docker Hub
1

I am using a docker based application which uses containers to provide microservices, Over a long run, my root filesystem is filed up.

Environment:

OS: Ubuntu 18.04.3 LTS server
Docker: Docker version 19.03.12, build 48a66213fe

Up on checking the main files which fills the disk space is /var/lib/docker/ , especially overlay2 directory.
I removed all stale docker images, did a system prune and still i am not able to freeup space.

root@ubuntu2:/home/test1# du -sch /var/lib/docker
4.3G /var/lib/docker
4.3G total
root@ubuntu2:/home/test1# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 395M 1.2M 394M 1% /run
/dev/mapper/ubuntu2–vg-root 14G 11G 2.4G 83% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/d4330d0d3bcee040588d204a8391761d87c56bd5ad6b6a8f3fac0c48fd0c51ce/merged
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/30ed1915a314fcd4cbeea7a9089f528e759bde07b1e8dd6eb30d7a647cb65266/merged
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/765167142da4fc17cf988ebfa7bf672a43266792aed252c75455e368caa680f6/merged
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/bb3fcf5eb730f23c40d8a60013c99c8844be6cdb37b8b98f67c01ac1b531b4cd/merged
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/84f3ebcb60fef50685a6f4dd4bfcf718f90f3e033c3c7ea5f7249adda07134ac/merged
shm 64M 0 64M 0% /var/lib/docker/containers/e61ce55f117e9be2528dca8053bcab1ecc6c2f5297840e0aa8dadfe3603ef548/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/4455c99beb21d04cf4daa42fdcf8494e65e419556a56ce9f99fbf6f06f054f93/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/916d2e317dec8d821c1605e331c1c8c2e48a081bee61c322a812ee3f9d7c86d1/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/53b44210516166800112fc9a3381f87efa1d7f2cb3f0d19c90165c32b8f3b587/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b256d9795dbc597f795ea3b4984bbfd2ab76b57d667b3608ff424c3e9d7c3f91/mounts/shm
overlay 14G 11G 2.4G 83% /var/lib/docker/overlay2/a2bc956d174f1e84becea809d036a932cd96e434f886eea6fb23abf50ec31b44/merged
shm 64M 0 64M 0% /var/lib/docker/containers/dc32e0e4788b2e55d3402e4892fe4dfab7a1abd9fe57ccf0420dba8a0161f44c/mounts/shm
/dev/vdb1 20G 45M 19G 1% /mnt/backup
tmpfs 395M 0 395M 0% /run/user/1001

2

Please add -x to your du command. It will prevent that the virtual overlayfs files will be counted additionaly.
Thus, in your case the occupied capacity for /var/lib/docker should be even less than the 4.3G you got as a result. The whole ‘df’ output is worth nothing, as it repeatedly shows the stats of /dev/mapper/ubuntu2–vg-root for different overlay mounts and shm memory mounts.

This brings us to the next point. If your block device has 14G, from which 11G are used, but /var/lib/docker itself only occupies 4.3G… Can you share what exactly made you conclude that docker would be the reason your harddisk is filled?

3

root@ubuntu2:/var/lib/docker# du -sch * |grep G
4.3G overlay2
4.3G total
root@ubuntu2:/var/lib/docker# cd overlay2/
root@ubuntu2:/var/lib/docker/overlay2# du -sch * |grep M
30M 020dc519a5ced58fbe9e8e2841ad08f48eaef5d92ca5a853e19316d33abee162
28M 0c3cbc06b5564e3d88d73f053a02e5f25eeffe4a824f0dcf5d840c3e082e8bc0
28M 0ece04b23506b92e7f65f54e82ecef9f07b91043ce60d2f6d8216ecd83688a3d
74M 0fc801fe7a3461ceda4bfefe10f445e062bc4458d93de3823c9906f0d1b4d2a4
7.2M 212c489a9851069b6af4be3c6a1985f2dcaeec96917153cd052dad7d7357cd2c
7.0M 21b7cecc4db99ed1d2b7876f01dc3c6322820bbe49f1128008ca0ffd17b16916
37M 223607bc7851c83aef0a2ffd4c7aed5398c11d0df1853d305ecb455a0034baf4
45M 25b0685b563abc88d75df2ee4c0a79f16d685a7c700a38742d2f37041df9794d
73M 2f014d9ae6feaba4bc587d7b500de1e63fcc2952350341219ebcb5fd71bc916d
308M 30ed1915a314fcd4cbeea7a9089f528e759bde07b1e8dd6eb30d7a647cb65266
99M 339936f1ea899c8fb76cf12828cb8b8d1f4a8c66c0ee8b580c17472146f78a3e
2.4M 38efaa93222a33f819482080e0950292d3ab061396284554e24f23194d37e653
4.2M 3bb2e0622f6b8c66951442600af071ea0719682ad985de87fb2af98d261ea842
76M 406d8b08ebf15218cecdb8d1198490a22e7baff351e7c8acaf794641bb4089be
47M 4ace76a6fe7f7861bdeb6065b02062ba24b3531d04e6f16edbca906ee08658b6
45M 4bf199e36ed98f2b1001c0baaca4cce18d3f939008c9d020f89d4269837be4a6
1.7M 4d45dd15c08c8c3f6aade9e2ecba4755ed33edb2dc06bdd5374448e8d055d71b
45M 4dad3e62e90f2a6adff2be0944071cf12cd0dad10c1415f74218af8c4fc8a36c
2.0M 5aac1b8ce1c15e518e428a848cd87f78ef9149fbb443661a2daa0a08ef3414f5
1.3M 5ee580c1438a9251ab3faef5644c90c9cdde7ddb155ba7c3cdb518cc3aadd5d3
326M 63d9833680dcd732db699c6eb5c46bda77e53bc731412980607b396dcd09aca4
46M 64ee7ef7ec3bff40808e6bfde2a21e0134871584c02cb6f1922dcf1ab202e8ff
47M 703cf046d531fd25c97af0a0924a79e4454641838ff98fb24f62018b6b1532cc
24M 73c664da4063e55338abd9192f09acc8f0fee8b2293f3318855d672c09d271cd
105M 765167142da4fc17cf988ebfa7bf672a43266792aed252c75455e368caa680f6
63M 78b3a3378907b47e3c89235019c7fbdade7f9a1feeb578da8d257ba32a713ca9
438M 84f3ebcb60fef50685a6f4dd4bfcf718f90f3e033c3c7ea5f7249adda07134ac
5.9M 8986b8466083398936d4bf113532a5a1a5bb22b57c246fc28d487e4dec4d595a
28M 89d1c9fb6abb68c47e4a9132f6011470a6de4280ce58b2ada1ac03bf1c87785d
20M 926db153906b9a1312241e747d37bfcd0b5f0255361ebd50aa63c51527e02762
64M 93c2a8e819bbf02694968257ffaeca98b9026e2c69bd47e088b074236b2a993a
46M 99df33c4228c5683f2057acd417e54f56700eb8d4d6f947613b186463102f05d
496M a2bc956d174f1e84becea809d036a932cd96e434f886eea6fb23abf50ec31b44
74M a7e8d2b4a5513cd20335a34918503fe98e2e9dc94ea85abefc0cfe14bc1d8f7c
63M af4793ac6c6b016e654f502a44e39544e9459ada650fd25d460f3905d14a149b
69M b27f41aad77212892adb20dc017c30e1bd8eecbb9946b53ccc3ff3024842ee27
16M baea26faf8369a4b1d1641cbb40c2d2b613c6e2b0d6c53a5633ceb613d6bd292
154M bb3fcf5eb730f23c40d8a60013c99c8844be6cdb37b8b98f67c01ac1b531b4cd
2.5M be924182bda9a2f8edcb44f89934fa6496591df637cbfda94aecde2118c3cb34
45M c14fe664164df05dc5316de5438c51c82bf90e364f17cb60afe7a0d624f5b0af
312M c23c88a16f092ef9f05467cfd2423e3fa60797b92125b55e3b641b669fb2a093
2.4M c63930962bacdd9505c5e68811f209feae4fd8ed01c0a59959ba5e5a72be6f1c
23M ca5459c18344e945b44fd863b71d8d5f6417f3d658bb8787fbbc1e570e5f6c36
2.2M d0a6f0f0c142ee4e0e2ca850a7371ef3c7ae3f6b09e57a3d5d9edb04d5a65b05
711M d4330d0d3bcee040588d204a8391761d87c56bd5ad6b6a8f3fac0c48fd0c51ce
17M d898f5c8682a09641d33a63ae300aa22b5ccbd2385cc49226da5868d3d2da9fd
2.3M e343dfbb25b6376ff82eba59c09e037c0ffba90e0740f700ac1dfa8d77a837cc
74M ed2cc633dcfd09fbe9807f9e9c3167c2379d14889da78f868a64bcbde6771136
2.5M ede14c2faa9698a3863b7cc54eab411cacd7b9a0a0b137bc39edeb213e57c6c9
64M f053d0318d1c28a890edf4be7b7b9c85564ee8f4a4206aa30779dd37d77c4d3a
23M f0e8de75e255ca27ac263b9b474b788804afd8447a7cf38593b286f1cf70f4d2
root@ubuntu2:/var/lib/docker/overlay2#

4

I have only 6 images… why so many file in overlay2 directory

5

the df command o/p made me think there are duplicate file since its showing 11G occupied by overlay files. Also i can see so many directories inside overlay2 . What can be the reason.

6

Like I pointed out in my earlier response: your du messurement was and still is wrong…

Let me ask again: how can a 4.3G folder (which will be even less if you messure right) be responsible to fill up a 14G partition?!

May I suggest this free excellent self paced docker traing? After the training you should have at least a decent understanding of how docker works and how things work under the hood.

Good luck!

7

Ok. Now its showing correct values. It has come down to 7G from 11G in df output. But i did not delete any files.
My point was different.
The df o/p is misleading since its showing different overlay2 directories.

8

root@ubuntu2:/# df -h
Filesystem Size Used Avail Use% Mounted on
udev 1.9G 0 1.9G 0% /dev
tmpfs 395M 1.2M 394M 1% /run
/dev/mapper/ubuntu2–vg-root 14G 7.0G 6.1G 54% /
tmpfs 2.0G 0 2.0G 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 2.0G 0 2.0G 0% /sys/fs/cgroup
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/d4330d0d3bcee040588d204a8391761d87c56bd5ad6b6a8f3fac0c48fd0c51ce/merged
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/30ed1915a314fcd4cbeea7a9089f528e759bde07b1e8dd6eb30d7a647cb65266/merged
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/765167142da4fc17cf988ebfa7bf672a43266792aed252c75455e368caa680f6/merged
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/bb3fcf5eb730f23c40d8a60013c99c8844be6cdb37b8b98f67c01ac1b531b4cd/merged
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/84f3ebcb60fef50685a6f4dd4bfcf718f90f3e033c3c7ea5f7249adda07134ac/merged
shm 64M 0 64M 0% /var/lib/docker/containers/e61ce55f117e9be2528dca8053bcab1ecc6c2f5297840e0aa8dadfe3603ef548/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/4455c99beb21d04cf4daa42fdcf8494e65e419556a56ce9f99fbf6f06f054f93/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/916d2e317dec8d821c1605e331c1c8c2e48a081bee61c322a812ee3f9d7c86d1/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/53b44210516166800112fc9a3381f87efa1d7f2cb3f0d19c90165c32b8f3b587/mounts/shm
shm 64M 0 64M 0% /var/lib/docker/containers/b256d9795dbc597f795ea3b4984bbfd2ab76b57d667b3608ff424c3e9d7c3f91/mounts/shm
overlay 14G 7.0G 6.1G 54% /var/lib/docker/overlay2/a2bc956d174f1e84becea809d036a932cd96e434f886eea6fb23abf50ec31b44/merged
shm 64M 0 64M 0% /var/lib/docker/containers/dc32e0e4788b2e55d3402e4892fe4dfab7a1abd9fe57ccf0420dba8a0161f44c/mounts/shm
/dev/vdb1 20G 45M 19G 1% /mnt/backup
tmpfs 395M 0 395M 0% /run/user/1001
root@ubuntu2:/#
root@ubuntu2:/# man du
root@ubuntu2:/#
root@ubuntu2:/# du -xsch * | grep G
2.4G home
du: cannot access ‘proc/8079/task/8079/fd/4’: No such file or directory
du: cannot access ‘proc/8079/task/8079/fdinfo/4’: No such file or directory
du: cannot access ‘proc/8079/fd/3’: No such file or directory
du: cannot access ‘proc/8079/fdinfo/3’: No such file or directory
1.3G usr
2.7G var
7.0G total