Rmi not freeing disk space in devicemapper sparse file (CentOS 6.6)

Docker Engine General
1

Hi folks,

On various versions of CentOS, my colleagues and I are running into the issue under discussion here, in which running “rmi” does not cause disk space to be freed from the devicemapper Docker storage file:
– https://github.com/docker/docker/issues/3182

Some terminal output showing the problem, from a fresh CentOS 6.6 VM and fresh install of Docker (see specific versions of everything further down):

-bash-4.1$ sudo du -h /var/lib/docker
…
292M /var/lib/docker/devicemapper
300M /var/lib/docker

-bash-4.1$ sudo docker pull centos
Pulling repository centos
fd44297e2ddb: Download complete
6941bfcbbfca: Download complete
41459f052977: Download complete
Status: Downloaded newer image for centos:latest

-bash-4.1$ sudo du -h /var/lib/docker
…
541M /var/lib/docker/devicemapper
548M /var/lib/docker

-bash-4.1$ sudo docker rmi centos
Untagged: centos:latest
-bash-4.1$ sudo docker rmi centos:7
Untagged: centos:7
-bash-4.1$ sudo docker rmi centos:centos7
Untagged: centos:centos7
Deleted: fd44297e2ddb050ec4fa9752b7a4e3a8439061991886e2091e7c1f007c906d75
Deleted: 41459f052977938b824dd011e1f2bec2cb4d133dfc7e1aa0e90f7c5d337ca9c4
Deleted: 6941bfcbbfca7f4f48becd38f2639157042b5cf9ab8c080f1d8b6d047380ecfc

-bash-4.1$ sudo du -h /var/lib/docker
…
541M /var/lib/docker/devicemapper
548M /var/lib/docker

Digging deep into the issue thread, it is suggested that upgrading to certain CentOS versions or kernel versions should bring in the necessary changes to resolve the problem, but so far we have not had success. A colleague of mine is seeing the same results from a fresh CentOS 7 VM and Docker 1.6 install.

Here are the versions of relevant components from the VM used in the test above:

-bash-4.1$ cat /etc/redhat-release
CentOS release 6.6 (Final)

-bash-4.1$ uname -a
Linux zzz.zzz.zzz 2.6.32-504.16.2.el6.x86_64 #1 SMP Wed Apr 22 06:48:29 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

-bash-4.1$ sudo docker info
Containers: 0
Images: 0
Storage Driver: devicemapper
Pool Name: docker-253:2-505066-pool
Pool Blocksize: 65.54 kB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 305.7 MB
Data Space Total: 107.4 GB
Metadata Space Used: 733.2 kB
Metadata Space Total: 2.147 GB
Udev Sync Supported: true
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.89-RHEL6 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 2.6.32-504.16.2.el6.x86_64
Operating System:
CPUs: 4
Total Memory: 7.688 GiB
Name: zzz.zzz.zzz
ID: ZFB3:AS7T:WPJW:ESLF:4LGQ:ELCB:W2UZ:ORLA:XAHL:7GF3:QU2Q:35EO

-bash-4.1$ sudo /sbin/dmsetup targets
thin-pool v1.14.0
thin v1.14.0
mirror v1.12.0
striped v1.5.6
linear v1.1.0
error v1.2.0

-bash-4.1$ sudo /sbin/dmsetup status
docker-253:2-505066-pool: 0 209715200 thin-pool 1 179/524288 4664/1638400 - rw no_discard_passdown queue_if_no_space

NOTE: I am suspicious of the “no_discard_passdown” setting shown in the dmsetup status output, but have so far not been able to find any documentation on how to change it or what to change it to…

Any advice on additional OS/component upgrades or Docker/devicemapper configuration settings that might resolve things?

Cheers.

2

Anecdotally, I have had no luck with devicemapper even with Docker 1.5. It keeps corrupting and producing weird impossible to debug states.

Is AUFS an option for you?

3

I was under the impression that AUFS was not preferred since it is not expected to become a standard available component in Fedora linux distributions (http://developerblog.redhat.com/2014/09/30/overview-storage-scalability-docker/).

Switching the storage type we configure Docker to use is certainly an option for us, as long as the alternative(s) address the problem and are stable and readily available for CentOS. This is for our production environment.

The frustrating thing is that others appear to have had success resolving the issue by upgrading (https://github.com/docker/docker/issues/3182#issuecomment-70704879), but we are still seeing the problem behavior on CentOS 6.6 and CentOS 7.

4

Some context here:

We have had enormous amounts of issues with devicemapper, so many that we are blacklisting it

Today we had a mini chat on twitter with @solomon who said:

@crosbymichael recommended trying out overlay

However there are still some know issues with extremely high inode count on overlay which are stopping it from becomming default or even high in the list of recommendations.

As of today the IDEAL and probably only perfectly working backend for Docker is AUFS, it will never find itself into the kernel, which is a pain.

My priority list of stuff I would try are

  1. aufs
  2. overlay (provided I do not have large amounts of images and layers)
  3. zfs
  4. btrfs

anything else is just not going to work.

1 Like