Docker compose and zfs volumes autocreation

Open Source Projects Compose
1

I’ve patched docker-zfs-plugin so I can provide only name to docker volume create, hoping to achieve compatibility with docker compose

Here is part of my config

version: "3.9"

volumes:
  postgres_16:
    driver: zfs

services:
  postgres_16:
    image: postgres
    container_name: postgres_16
    ports:
      - 5432:5432
    volumes:
      - "./configs/postgresql/postgresql.conf:/var/lib/postgresql/data/postgresql.conf"
      - "/etc/localtime:/etc/localtime"
      # Note that postgres image assumes /var/lib/postgresql/data as a volume mount point!
      - "postgres_16:/var/lib/postgresql/data"

docker compose up -d postgres

results in an error

external volume “stagingc_postgres_16” not found

Isn’t docker compose supposed to create it automatically? I can use external: true and create all volumes manually but that’s not what I’ve expected

2

Yes, compose would create a volume automatically, unless it is an external volume. Which would require the “external: true” option to be set. Since you mentioned you patched the zfs volume plugin, this behaviour could be caused by the patch or the original plugin had the bug already.

3

Volume I’ve described is not intended to be external. The only thing I’ve patched is mangling/demangling of name, what plugin call should I trace to find the reason? What does docker compose expect from the plugin? I see the only call

Dec  1 15:12:35 localhost docker-zfs-plugin[92069]: time="2023-12-01T15:12:35+03:00" level=debug msg=Get Request="&{stagingc_postgres_16}"

so it tries to find volume, fails to find and… somehow decides that the volume should be external?

4

I understood that you didn’t want to create external volumes, that’s why I wrote it could be a bug in the original plugin, but I never tried the zfs plugin or try to develop / patch any plugin so I can’t help you with that.

5

Are there any description of docker compose interactions with plugins?

6

I doubt that Docker Compose has anything to do with plugins. Docker Compose is mainly a yaml based client for Docker. This is what I found in the documentation

https://docs.docker.com/engine/extend/plugins_volume/

It starts with a changelog which almost fooled me, but if you scroll down there is more.

And this is about developing a plugin which I assume you already know better then I do.

https://docs.docker.com/engine/extend/#developing-a-plugin

7

I have no idea how the plugin or your modification works, but I am still going to share some details about compose and volumes.

By default, a volume managed by the compose project is prefixed with {project name}_ when created. If no project name is provided, then the directory name will be used as project name. In your case the compose file seem to be in a folder named stagingc. If you don’t like the volume name to be prefixed with {project name}_, you can override it with name: postgres_16 (this does not require external: true).

8

I assume you try to use this volume plugin: GitHub - TrilliumIT/docker-zfs-plugin: Docker volume plugin for creating persistent volumes as a dedicated zfs dataset.

I looked at the systemd unit and saw the argument --dataset-name tank/docker-volumes.

Then I saw the example command to create a volume: docker volume create -d zfs -o compression=lz4 -o dedup=on --name=tank/docker-volumes/data.

Is it safe to assume that the problem is that the dataset name must be the prefix of the volume name, in order to be created on that dataset? If so, then declaring the volume with name: tank/docker-volumes/data might work.

1 Like
9

I’ve patched the driver so I need to specify only the last part of ZFS fs name. The problem is that docker does not ask to create volume at all, while it should. Okay, I have docker sources…