Docker Community Forums

Share and learn in the Docker community.

Docker on ZFS - Change Parent Dataset

Hello, as it took me a while to find a solution to this problem, I though I might post here in case someone else happens to struggle with a similar situation :wink:

My dev environment is based on Debian 10 with a ZFS root filesystem located on a pool (rpool) whose layout is as follows:

$ sudo zfs list -r rpool
NAME                             USED  AVAIL  REFER  MOUNTPOINT
rpool                            247G   652G    96K  /
rpool/ROOT                      40.2G   652G    96K  none
rpool/ROOT/debian-10            40.2G   652G  10.5G  /
rpool/ROOT/debian-10/opt        4.36G   652G  1.16G  /opt
rpool/ROOT/debian-10/var        11.1G   652G    96K  /var
rpool/ROOT/debian-10/var/cache  11.0G   652G  11.0G  /var/cache
rpool/ROOT/debian-10/var/log    18.5M   652G  18.5M  legacy
rpool/ROOT/debian-10/var/tmp    47.4M   652G  47.4M  legacy
rpool/home                       139G   652G  92.4G  /home
rpool/home/root                 1.93M   652G   948K  /root
rpool/swap                      68.0G   715G  4.60G  -
rpool/tmp                        620K   652G   620K  legacy

When I started using Docker, I discovered that it was creating a lot of datasets right under the root dataset:

$ sudo zfs list -r rpool
NAME                                  USED  AVAIL  REFER  MOUNTPOINT
rpool                                 246G   653G    96K  /
rpool/ROOT                           39.2G   653G    96K  none
rpool/ROOT/debian-10                 39.2G   653G  10.3G  /
rpool/ROOT/debian-10/0594a....       5.04M   653G  53.9M  legacy
rpool/ROOT/debian-10/094e0....        560K   653G  56.0M  legacy
rpool/ROOT/debian-10/0a248....        200K   653G  44.9M  legacy
rpool/ROOT/debian-10/12dc5....        568K   653G  56.0M  legacy
rpool/ROOT/debian-10/17829....        108K   653G  56.0M  legacy
rpool/ROOT/debian-10/1a6ce....        160K   653G  78.3M  legacy
rpool/ROOT/debian-10/1c27c....       1.62M   653G  55.4M  legacy
rpool/ROOT/debian-10/20f13....        144K   653G  57.6M  legacy
rpool/ROOT/debian-10/254ed....       44.8M   653G  44.8M  legacy
rpool/ROOT/debian-10/2997e....       7.79M   653G  51.9M  legacy
rpool/ROOT/debian-10/2e725....       20.9M   653G  76.4M  legacy
...
...

This is obviously quite annoying and it would be much preferable to have all Docker related datasets grouped under a dedicated one.

I immediately jumped to the Docker documentation, which BTW is really well done, and found the [Docker ZFS Storage page] (docs.docker .com/storage/storagedriver/zfs-driver/) giving a lot of detailed explanations about ZFS itself and stating that creating a dataset that gets mounted under /var/lib/docker should be the solution.
I did apply the recommendation, verifying that the ZFS dataset was well mounted (/var/lib/docker has to be empty!), only to discover that, unfortunately, even though I followed the procedure, Docker still referenced the root dataset as its Parent Dataset:

$ sudo zfs list -r rpool
NAME                           USED  AVAIL  REFER  MOUNTPOINT
rpool                          246G   653G    96K  /
rpool/ROOT                    39.2G   653G    96K  none
rpool/ROOT/debian-10          39.2G   653G  10.3G  /
rpool/ROOT/debian-10/opt      4.36G   652G  1.16G  /opt
rpool/ROOT/debian-10/var      11.1G   652G    96K  /var
rpool/ROOT/debian-10/var/cache 11.0G   652G  11.0G  /var/cache
rpool/ROOT/debian-10/var/log  18.5M   652G  18.5M  legacy
rpool/ROOT/debian-10/var/tmp  47.4M   652G  47.4M  legacy
rpool/docker                  96K   652G    96K  /var/lib/docker
...

$ docker info
Client:
 Debug Mode: false

Server:
 ...
 Server Version: 19.03.9
 Storage Driver: zfs
  Zpool: rpool
  Zpool Health: ONLINE
  Parent Dataset: rpool/ROOT/debian-10
 ...
 Kernel Version: 4.19.0-9-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 ...
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 ...

Digging deeper in the Docker documentation I finally found out that the docker daemon can be configured using an /etc/docker/daemon.json file (as described here), while all the possible options are listed on the dockerd page

Thus, to specify the Parent Dataset to be used by the Docker daemon for its own storage, one needs to first stop the docker service, create/modify the /etc/docker/daemon.json file content and then start the docker daemon again, as follows:

$ sudo systemctl stop docker
$ sudo nano /etc/docker/daemon.json

{
  "storage-opts": [ "zfs.fsname=rpool/docker"]
}

$ sudo systemctl start docker

And now, docker will create its legacy datasets at the desired location:

$ docker info
...
  Parent Dataset: rpool/docker

$ sudo zfs list -r rpool/docker
NAME                         USED  AVAIL  REFER  MOUNTPOINT
rpool/docker                 287M   652G    96K  /var/lib/docker
rpool/docker/0225c9b...      192K   652G  44.9M  legacy
rpool/docker/0e3f3f5...     20.8M   652G  64.6M  legacy
rpool/docker/28b0bc9...      144K   652G   178M  legacy
rpool/docker/2ee79f0...     1.62M   652G  43.7M  legacy
...
...

Hopefully this information will be useful to others!

1 Like

I have change data set just like your methos.