Mount reports wrong volumes when using devicemapper

Hi,

just installed an evaluation version of 17.06.2-ee-6 on a freshly installed OL 7.3 host:

[ns@dummy ~]$ uname -a
Linux dummy 3.10.0-514.26.2.el7.x86_64 #1 SMP Tue Jul 4 07:15:04 PDT 2017 x86_64 x86_64 x86_64 GNU/Linux

[ns@dummy ~]$ cat /etc/oracle-release
Oracle Linux Server release 7.3

[ns@dummy ~]$ docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.06.2-ee-6
Storage Driver: devicemapper
 Pool Name: vg01-lv_docker_thinpool
 Pool Blocksize: 524.3kB
 Base Device Size: 10.74GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 19.92MB
 Data Space Total: 42.95GB
 Data Space Available: 42.93GB
 Metadata Space Used: 389.1kB
 Metadata Space Total: 5.369GB
 Metadata Space Available: 5.368GB
 Thin Pool Minimum Free Space: 4.295GB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
 seccomp
  Profile: default
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: Oracle Linux Server 7.3
OSType: linux
Architecture: x86_64
CPUs: 24
Total Memory: 15.37GiB
Name: dummy
ID: VGIU:U5GG:GEMH:MGJR:4X6Q:X4HV:DL4F:BV6M:JI3B:2IWQ:RZQI:KDBK
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

The system has two volume groups:

[root@dummy /]# vgdisplay
  --- Volume group ---
  VG Name               vg00
  System ID
  Format                lvm2
  Metadata Areas        1
  Metadata Sequence No  10
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                9
  Open LV               9
  Max PV                0
  Cur PV                1
  Act PV                1
  VG Size               135.62 GiB
  PE Size               4.00 MiB
  Total PE              34719
  Alloc PE / Size       13824 / 54.00 GiB
  Free  PE / Size       20895 / 81.62 GiB
  VG UUID               vuaDZe-puSm-OI15-Nez4-lPT5-ycZp-tPHWex

  --- Volume group ---
  VG Name               vg01
  System ID
  Format                lvm2
  Metadata Areas        2
  Metadata Sequence No  35
  VG Access             read/write
  VG Status             resizable
  MAX LV                0
  Cur LV                7
  Open LV               5
  Max PV                0
  Cur PV                2
  Act PV                2
  VG Size               557.73 GiB
  PE Size               4.00 MiB
  Total PE              142780
  Alloc PE / Size       122112 / 477.00 GiB
  Free  PE / Size       20668 / 80.73 GiB
  VG UUID               TkLgEy-UACf-q2bG-hTEQ-Dep1-vXIl-I3LLC8

Here’s what I did to create the docker thinpool:

  • lvcreate --wipesignatures y -n lv_docker_thinpool vg01 -L 40G
  • lvcreate --wipesignatures y -n lv_docker_thinpoolmeta vg01 -L 5G
  • lvconvert -y --zero n -c 512K --thinpool vg01/lv_docker_thinpool --poolmetadata vg01/lv_docker_thinpoolmeta
  • lvchange --metadataprofile docker-thinpool vg01/lv_docker-thinpool
  • lvs -o+seg_monitor

Here the daemon.json:

{
        "storage-driver": "devicemapper",
        "storage-opts":
        [
                "dm.thinpooldev=/dev/mapper/vg01-lv_docker_thinpool",
                "dm.use_deferred_removal=true"
        ]
}

Docker-Engine is able to start without any problems.

As per the documentation (c.f. https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#how-the-devicemapper-storage-driver-works) I should be able to see the mounted thinpool when docker engine is running, e.g.:

$ mount |grep devicemapper
/dev/xvda1 on /var/lib/docker/devicemapper type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

But here’s what happens on my system:

[root@dummy/]# mount | grep docker
/dev/mapper/vg00-lv_var on /var/lib/docker/plugins type ext4 (rw,relatime,data=ordered)
/dev/mapper/vg00-lv_var on /var/lib/docker/devicemapper type ext4 (rw,relatime,data=ordered)

That is the wrong volume group and the wrong logical volume as well…
Furthermore, docker info reports that the backing filesystem is xfs, not ext4.
Since everything is running “fine”, I expect this to be just a display problem - but where should I start debugging?

Here is an excerpt of the other lvs that exist on the machine:

[root@dummy /]# lsblk
NAME                              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                                 8:0    0 136.1G  0 disk
├─sda1                              8:1    0   512M  0 part /boot
└─sda2                              8:2    0 135.6G  0 part
  ├─vg00-lv_root                  253:0    0     2G  0 lvm  /
  ├─vg00-lv_swap                  253:1    0    16G  0 lvm  [SWAP]
  ├─vg00-lv_usr                   253:2    0     6G  0 lvm  /usr
  ├─vg00-lv_home                  253:5    0     2G  0 lvm  /home
  ├─vg00-lv_var                   253:6    0    16G  0 lvm  /var
  ├─vg00-lv_tmp                   253:7    0     3G  0 lvm  /tmp
  └─vg00-lv_opt                   253:8    0     4G  0 lvm  /opt
sdb                                 8:16   0 278.9G  0 disk
└─sdb1                              8:17   0 278.9G  0 part
  └─vg01-lv_app                   253:9    0   300G  0 lvm  /app
sdc                                 8:32   0 278.9G  0 disk
└─sdc1                              8:33   0 278.9G  0 part
  ├─vg01-lv_app                   253:9    0   300G  0 lvm  /app
  ├─vg01-lv_docker_thinpool_tmeta 253:15   0     5G  0 lvm
  │ └─vg01-lv_docker_thinpool     253:17   0    40G  0 lvm
  └─vg01-lv_docker_thinpool_tdata 253:16   0    40G  0 lvm
    └─vg01-lv_docker_thinpool     253:17   0    40G  0 lvm
sr0                                11:0    1  1024M  0 rom

So the mount suggests that /var/lib/docker/plugins and /var/lib/docker/devicemapper are a mount of the same (wrong) logical volume.
That is, of course, not true:

[root@dummy/]# ls -l /var/lib/docker/plugins
total 12
drwx------ 4 root root 4096 Dec  7 18:26 b7ee60877053565a5d5807ec497a49bd4f2195860d4c4f25d7049bd54fe4b647
drwx------ 3 root root 4096 Dec  7 16:21 storage
drwx------ 2 root root 4096 Dec  7 16:38 tmp
[root@dummy/]# ls -l /var/lib/docker/devicemapper
total 8
drwx------ 2 root root 4096 Dec  7 18:26 metadata
drwxr-xr-x 2 root root 4096 Dec  7 17:17 mnt

Any insight is very appreciated…

Greetings

Nico