Hello !
(I also posted on r/docker but I did not get an answer so I’m also trying here 
)
I improved an existing docker volume plugin to create ZFS datasets for each volume. I’m giving some background on why & how below.
Background
I’m trying to modernize the infrastructure of a non profit I give some of my time to and we should be able to move everything to docker with a CI to build & deploy our custom containers. Which is great so I moved on the task of building the core blocks of the new infrastructure, and one the them is backups & snapshots of the containers state.
We want good backups and one way to achieve that is to stop the container, backup its state and start it again. We’re pretty tolerant to a few minutes of downtime each day during specific hours so that sounds good in theory, but for containers with a lot of state, backups could take a long time. One solution to this problem is to stop the container, take a snapshot of its state, start it again and backup the snapshot. That’s what we already do for some services backed by ZFS and I’d like to that with docker too.
When you’re doing something that may explode unpredictably, like an update on a customized installation of some software, you may also want to be able to rollback easily. Snapshots are a great way to achieve that.
Having a different dataset for each volume also make possible to enable compression or not (or other ZFS features) on a more granular level for example. I always enable compression because it’s almost free in terms of compute and can save A LOT of space.
Plugin
https://github.com/icefo/docker-zfs-plugin
The smelly workaround
It was forked from someone that ported the original plugin to V2 plugin API. While it sounds great, he had to resort to a hack to make it work. To understand here is some background on the v2 volume plugin architecture:
In short volume plugins are containers and have to mount the volumes in a specific folder in the container that is shared with the host. Sounds great in theory but ZFS is a kernel level driver, so the mountpoints will be relative to the host and not the container (this break the encapsulation). The workaround he found is:
- Define this path as the shared folder:
/var/lib/docker/plugins/pluginHash/propagated-mount/ - Add this
../../../../../..to all paths returned to docker from the plugin to return to the true host root path
This allows the plugin to mount the ZFS datasets wherever it wants in the system (he chose the docker volumes folder, the doc specifically tells to not do that, so I defined an other) while making the docker engine happy.
My question is: How likely is this to break in some future version of docker ? I’m sure this is unintended from the docker developers and is also probably a security vulnerability. From a security point of view it’s probably not too bad because most plugins seem to have the cap CAP_SYS_ADMIN anyway.
Does this plugin interest someone ?
After ‘completing’ the plugin, I realized I could at least achieve the fast backup functionality I wanted by making mounting a ZFS dataset in /var/lib/docker/volumes. I would lose the fast snapshot & restore functionality for individual volumes but I would be much less likely to break after a docker engine update.
The plugin works but it needs some polish to be usable in production (for example, it logs events to a hard coded folder)
Do you think there is a way to improve this plugin ?
Like obviously, it’s more a proof of concept. I’m thinking about a way to do it without the smelly workaround. I’d like to implement that directly on the docker engine if it’s possible with a medium effort, but I have no idea of the architecture of the docker engine and I don’t know if it would be welcome.