How to have /var/lib/docker on a 100G drive?

General
1

Hi all,

I created a new VM and added a new partition (100G) so I can allocate it to docker later. It has the following partitions and disks:

Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 30G 3.8G 27G 13% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.7G 0 7.7G 0% /dev/shm
tmpfs 7.7G 8.4M 7.7G 1% /run
tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup
/dev/xvdg1 30G 33M 30G 1% /home
/dev/xvdb1 30G 552M 30G 2% /var
/dev/xvde1 30G 33M 30G 1% /var/tmp
/dev/xvdc1 30G 33M 30G 1% /var/log
/dev/xvdd1 30G 33M 30G 1% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1000

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 30G 0 part /
xvdb 202:16 0 30G 0 disk
└─xvdb1 202:17 0 30G 0 part /var
xvdc 202:32 0 30G 0 disk
└─xvdc1 202:33 0 30G 0 part /var/log
xvdd 202:48 0 30G 0 disk
└─xvdd1 202:49 0 30G 0 part /var/log/audit
xvde 202:64 0 30G 0 disk
└─xvde1 202:65 0 30G 0 part /var/tmp
xvdf 202:80 0 100G 0 disk
└─xvdf1 202:81 0 30G 0 part
xvdg 202:96 0 30G 0 disk
└─xvdg1 202:97 0 30G 0 part /home

Upon installing Docker, I noticed docker did not use that xvdf device and I can see the free device as well as the loopback device created by Docker’s defaults:, it added a new 100G in there:

Filesystem Size Used Avail Use% Mounted on
/dev/xvda2 30G 3.8G 27G 13% /
devtmpfs 7.8G 0 7.8G 0% /dev
tmpfs 7.7G 0 7.7G 0% /dev/shm
tmpfs 7.7G 8.4M 7.7G 1% /run
tmpfs 7.7G 0 7.7G 0% /sys/fs/cgroup
/dev/xvdg1 30G 33M 30G 1% /home
/dev/xvdb1 30G 552M 30G 2% /var
/dev/xvde1 30G 33M 30G 1% /var/tmp
/dev/xvdc1 30G 33M 30G 1% /var/log
/dev/xvdd1 30G 33M 30G 1% /var/log/audit
tmpfs 1.6G 0 1.6G 0% /run/user/1000

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 30G 0 part /
xvdb 202:16 0 30G 0 disk
└─xvdb1 202:17 0 30G 0 part /var
xvdc 202:32 0 30G 0 disk
└─xvdc1 202:33 0 30G 0 part /var/log
xvdd 202:48 0 30G 0 disk
└─xvdd1 202:49 0 30G 0 part /var/log/audit
xvde 202:64 0 30G 0 disk
└─xvde1 202:65 0 30G 0 part /var/tmp
xvdf 202:80 0 100G 0 disk
└─xvdf1 202:81 0 30G 0 part
xvdg 202:96 0 30G 0 disk
└─xvdg1 202:97 0 30G 0 part /home
loop0 7:0 0 100G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm

I then created another VM and this time I did not create a new 100G disk. But then when I installed Docker on this VM, I noticed the same 100G pool was created:

NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 30G 0 part /
xvdb 202:16 0 30G 0 disk
└─xvdb1 202:17 0 30G 0 part /var
xvdc 202:32 0 30G 0 disk
└─xvdc1 202:33 0 30G 0 part /var/log
xvdd 202:48 0 30G 0 disk
└─xvdd1 202:49 0 30G 0 part /var/log/audit
xvde 202:64 0 30G 0 disk
└─xvde1 202:65 0 30G 0 part /var/tmp
xvdg 202:96 0 30G 0 disk
└─xvdg1 202:97 0 30G 0 part /home
loop0 7:0 0 100G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm

My question is how can I have a VM that has 100G allocated to docker so /var/lib/docker ends up on the 100G drive?

Thanks in advance

2

stop the docker daemon
move all the /var/lib/docker to a folder on the space you want
create a symbolic link for var/lib/docker to the new folder
start the docker daemon

3

Thanks for your reply. How exactly will I do that? I just created a brand new VM, the xvdf device is set to be 100G. But when I installed docker, it didn’t use the xvdf, instead, it created its own disk:

[root@ip-10-17-1-187 centos]# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda 202:0 0 30G 0 disk
├─xvda1 202:1 0 1M 0 part
└─xvda2 202:2 0 30G 0 part /
xvdb 202:16 0 30G 0 disk
└─xvdb1 202:17 0 30G 0 part /var
xvdc 202:32 0 30G 0 disk
└─xvdc1 202:33 0 30G 0 part /var/log
xvdd 202:48 0 30G 0 disk
└─xvdd1 202:49 0 30G 0 part /var/log/audit
xvde 202:64 0 30G 0 disk
└─xvde1 202:65 0 30G 0 part /var/tmp
xvdf 202:80 0 100G 0 disk
└─xvdf1 202:81 0 30G 0 part
xvdg 202:96 0 30G 0 disk
└─xvdg1 202:97 0 30G 0 part /home
loop0 7:0 0 100G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm
loop1 7:1 0 2G 0 loop
└─docker-202:17-100789143-pool 253:0 0 100G 0 dm

Thanks

4

the commands are

mv = move
ln = link

stop/start docker, read the online doc…

google search can help u find answers to just about every question… its what I do here if I don’t know.

5

I am familiar with the basic commands. The process to achieve what I was looking for is more advanced than that.

6

other have done the same thing without issue…

you cannot change WHERE docker look (/var/lib/docker) so you can only symlink it to the new place…

7

The -pool volumes are not disks, they’re virtual loopback devices backed by files in /var/lib/docker (/devicemapper?). Anyway, sdetweil already gave you all you needed:

systemctl stop docker
mv /var/lib/docker $xvdf_mountpoint
ln --symbolic $xvdf_mountpoint /var/lib/docker
systemctl start docker

(Note that it isn’t mounted anywhere yet per your lsblk output.)

I have never tried it, but the Docker engine should follow the link fine.

That being said, an even easier way would be to mount the xvdf device on /var/lib/docker directly.

systemctl stop docker
mount /dev/xvdf1 /var/lib/docker
systemctl start docker

(Also note that your only partition on xvdf is 30G, you might want to increase that. In fact, you don’t have to partition a device if you’re using it entirely for one thing – e.g. one filesystem.)

Again I have never done this though I’m quite confident it would work transparently. I do regularly mount /var/lib/docker/volumes on arbitrary devices (for application data).

8

thank for sharing, it works for me. But i am still puzzled is i need ln or mount for my production environment. what is the good practice to use /var/lib/docker on symbolic link or mount? what are known issues for both? it can probably help us to decide what option should consider.

9

Symlinking the folder is a dirty solution.

Stop the Docker service. Directly mount a filesystem (ext4, xfs with d_type true flag!) into /var/lib/docker from /etc/fstab or mount it somehwere else and bend the data-root configuration in /etc/docker/daemon.json (the file might not exist yet). Then restart the Docker service again.

Check the output of docker info | grep 'Storage Driver:' to see which storagedriver is used. On wrong formated xfs, this will fall back to device mapper. If the command returs “overlay2” everything is fine. If it says “device mapper” make sure to figure out why and remedy that problem.