ZFS error on Docker Engine 29

lb88 · November 11, 2025, 2:12pm

Hi, I updated to docker 29 today and instantly docker crashed. Checking journalctl revealed the following error:

failed to start daemon: configured driver "zfs" not available: unavailable

This is my daemon.json file:

{
  "storage-driver": "zfs",
  "data-root": "/mnt/apps/docker"
}

Any idea why this would have happened? I’ve downgraded to v28 for the time being and everything is working again as normal.

flofi · November 11, 2025, 6:54pm

Not gonna help much but just to confirm: same issue here. Rolling back to v28 fixed it as well.

Debian Bookworm on ZFS

lb88 · November 11, 2025, 6:58pm

That’s good to know, I was wondering whether to just got for a reinstall and see if it was a one-off but I guess not. I’m on Debian Trixie by the way.

nsmrtks · November 12, 2025, 12:53am

Same issue here.
Two Ubuntu 24.04.03 on x86_64 computers. One physical and one virtual machine.
Both have /var/lib/docker as a ZFS filesystem.

Docker Engine has been installed following the steps below.

/etc/docker/daemon.json

{
        "storage-driver": "zfs",
        "proxies": {
                ...
        }
}

I did ‘apt upgrade’ yesterday, docker-ce is upgraded, and it does not start.

/var/log/syslog says:

... dockerd[18843]: failed to start daemon: configured driver "zfs" not available: unavailable

I downgraded docker-ce, then it is working now.

problematic: 5:29.0.0-1~ubuntu.24.04~noble
downgraded: 5:28.5.2-1~ubuntu.24.04~noble

The below does not mention about ZFS.

API breaking changes?

kevinruffus · November 12, 2025, 1:24am

Ok, so it isn’t just me.

Ran updates, Docker took a dump.

Ubuntu 24.04

itris666 · November 12, 2025, 2:26am

same here. ubuntu noble as well. no related specific config that I could think of.

vinny8675309 · November 12, 2025, 6:14am

I’m seeing the same error as well.

2025-11-12T05:25:50.799472+00:00 HOST dockerd[4069105]: failed to start daemon: configured driver "zfs" not available: unavailable

I’m also running Noble:

> lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 24.04.3 LTS
Release:        24.04
Codename:       noble

Reverting to v28.5.2 got everything back running for me as well.

rimelek · November 12, 2025, 9:15pm

I checked the source code of moby and I see that this error message is only shown when the containerd image store is used. Which is by default enabled in the new version

containerd image store is now the default for fresh installs .

Although it says “fresh installs”, after upgrading, I got the new snapshotter as well. I am not sure if zfs should work with the new image store, but I disabled the containerd snapshotter in the docker daemon config

  "features": {
    "containerd-snapshotter": false
  }

Then I got a different error message, which is understandable as I have no zfs installed on my test machine.

level=info msg="[graphdriver] trying configured driver: zfs"
level=debug msg="zfs command is not available: exec: \"zfs\": executable file not found in $>

This is the source code by the way if someone is interested

github.com/moby/moby

daemon/daemon.go

d105562be


      
          		if driverName == "" {
          			if _, ok := availableDrivers[defaults.DefaultSnapshotter]; ok {
          				driverName = defaults.DefaultSnapshotter
          			} else if _, ok := availableDrivers["native"]; ok {
          				driverName = "native"
          			} else {
          				log.G(ctx).WithField("available", maps.Keys(availableDrivers)).Debug("Preferred snapshotter not available in containerd")
          				return nil, fmt.Errorf("snapshotter selection failed, no drivers available: %w", cerrdefs.ErrUnavailable)
          			}
          		} else if _, ok := availableDrivers[driverName]; !ok {
          			return nil, fmt.Errorf("configured driver %q not available: %w", driverName, cerrdefs.ErrUnavailable)
          		}
          
          		// Configure and validate the kernels security support. Note this is a Linux/FreeBSD
          		// operation only, so it is safe to pass *just* the runtime OS graphdriver.
          		if err := configureKernelSecuritySupport(&cfgStore.Config, driverName); err != nil {
          			return nil, err
          		}
          		d.usesSnapshotter = true
          		d.imageService = ctrd.NewService(ctrd.ImageServiceConfig{
          			Client:          d.containerdClient,

But if I intrpret it correctly, the storage driver is added to the chcked list only when the containerd snapshotter is used.

github.com/moby/moby

daemon/daemon.go

d105562be


      
          		resp, err := d.containerdClient.IntrospectionService().Plugins(ctx, `type=="io.containerd.snapshotter.v1"`)
          		if err != nil {
          			return nil, fmt.Errorf("failed to get containerd plugins: %w", err)
          		}
          		if resp == nil || len(resp.Plugins) == 0 {
          			return nil, fmt.Errorf("failed to get containerd plugins response: %w", cerrdefs.ErrUnavailable)
          		}
          		availableDrivers := map[string]struct{}{}
          		for _, p := range resp.Plugins {
          			if p == nil || p.Type != "io.containerd.snapshotter.v1" {
          				continue
          			}
          			if p.InitErr == nil {
          				availableDrivers[p.ID] = struct{}{}
          			} else if (p.ID == driverName) || (driverName == "" && p.ID == defaults.DefaultSnapshotter) {
          				log.G(ctx).WithField("message", p.InitErr.Message).Warn("Preferred snapshotter not available in containerd")
          			}
          		}
          
          		if driverName == "" {
          			if _, ok := availableDrivers[defaults.DefaultSnapshotter]; ok {

EDIT:

I found this issue on GitHub

github.com/moby/moby

ZFS backend broken in v29.0.0

opened 10:29AM - 11 Nov 25 UTC

CryptoCopter

status/0-triage kind/bug area/storage/zfs version/29.0

### Description After upgrading to `v29.0.0`, `dockerd` crashes on startup beca…use it can't load the ZFS storage backend driver. While [the release page](https://github.com/moby/moby/releases/tag/docker-v29.0.0) mentions several "breaking changes", removed ZFS support does not appear to be among them, so I assume this is a bug. ### Reproduce 1. configure [ZFS storage backend](https://docs.docker.com/engine/storage/drivers/zfs-driver/) 2. crash ### Expected behavior don't crash ### docker version ```bash Client: Docker Engine - Community Version: 29.0.0 API version: 1.52 Go version: go1.25.4 Git commit: 3d4129b Built: Mon Nov 10 21:46:31 2025 OS/Arch: linux/amd64 Context: default ``` ### docker info ```bash Client: Docker Engine - Community Version: 29.0.0 Context: default Debug Mode: false Plugins: buildx: Docker Buildx (Docker Inc.) Version: v0.29.1 Path: /usr/libexec/docker/cli-plugins/docker-buildx compose: Docker Compose (Docker Inc.) Version: v2.40.3 Path: /usr/libexec/docker/cli-plugins/docker-compose ``` ### Additional Info ``` Nov 11 11:14:42 dstank dockerd[4924]: time="2025-11-11T11:14:42.988491769+01:00" level=info msg="Starting up" Nov 11 11:14:42 dstank dockerd[4924]: time="2025-11-11T11:14:42.989649796+01:00" level=info msg="OTEL tracing is not configured, using no-op tracer provider" Nov 11 11:14:42 dstank dockerd[4924]: time="2025-11-11T11:14:42.989797369+01:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/etc/cdi Nov 11 11:14:42 dstank dockerd[4924]: time="2025-11-11T11:14:42.989826479+01:00" level=info msg="CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory" dir=/var/run/cdi Nov 11 11:14:42 dstank dockerd[4924]: time="2025-11-11T11:14:42.989942020+01:00" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so using resolv.conf: /run/systemd/resolve/resolv.conf" Nov 11 11:14:43 dstank dockerd[4924]: time="2025-11-11T11:14:43.007009507+01:00" level=info msg="Creating a containerd client" address=/run/containerd/containerd.sock timeout=1m0s Nov 11 11:14:43 dstank dockerd[4924]: time="2025-11-11T11:14:43.014541153+01:00" level=info msg="Loading containers: start." Nov 11 11:14:43 dstank dockerd[4924]: time="2025-11-11T11:14:43.016865083+01:00" level=info msg="Starting daemon with containerd snapshotter integration enabled" Nov 11 11:14:43 dstank dockerd[4924]: time="2025-11-11T11:14:43.019152647+01:00" level=warning msg="Preferred snapshotter not available in containerd" message="lstat /var/lib/containerd/io.containerd.snapshotter.v1.zfs: no such file or directory: skip plugin" Nov 11 11:14:43 dstank dockerd[4924]: failed to start daemon: configured driver "zfs" not available: unavailable Nov 11 11:14:43 dstank systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE Nov 11 11:14:43 dstank systemd[1]: docker.service: Failed with result 'exit-code'. Nov 11 11:14:43 dstank systemd[1]: Failed to start docker.service - Docker Application Container Engine. Nov 11 11:14:45 dstank systemd[1]: docker.service: Scheduled restart job, restart counter is at 3. Nov 11 11:14:45 dstank systemd[1]: docker.service: Start request repeated too quickly. Nov 11 11:14:45 dstank systemd[1]: docker.service: Failed with result 'exit-code'. Nov 11 11:14:45 dstank systemd[1]: Failed to start docker.service - Docker Application Container Engine. ```

itris666 · November 15, 2025, 5:14pm

There has just been an update in ubuntu noble repo, where “containerd-snapshotter” is again false by default. I’ve transitioned to it so I had to reactivate it in daemon.json;

{
  "storage-driver" : "zfs",
  "features":{
    "containerd-snapshotter" : true
  }
}

lb88 · November 15, 2025, 5:41pm

I saw the update as well - I bit the bullet and updated from 28 to 29.0.1 and zfs is working as normal.

nsmrtks · November 18, 2025, 12:15am

I upgraded docker-ce to 5:29.0.1-1~ubuntu.24.04~noble yesterday.
After upgrading, it starts, without any changes to ‘/etc/docker/daemon.json’.
And I have not encountered any problems for half a day.

Now docker-ce is upgraded to 5:29.0.2-1~ubuntu.24.04~noble this morning.
I hope nothing goes wrong with this version also.

meyay · November 18, 2025, 7:59am

For all those that haven’t read the Docker Engine v29 blog post, there are two mentions of the graph driver that will affect you:

Legacy graph drivers are still available, but are now deprecated. New installs can still opt out of Containerd image store if there is any issue.

The graph driver backend will be removed in a future release.

So from docker v30 on a config like @itris666 shared might become necessary.

mastercatz · November 30, 2025, 10:33am

oh great so I wasted a day because of a regression

practically nuked my OS trying to reinstall docker to find it was just zfs that was not working

pointing finger at docker-rootless as it seems to use fuse-overlayfs regardless what setting is used

also might swap over to a dedicated zfs dataset / volume and try

{
“storage-opts”: [ “zfs.fsname=ZFS3WAY24B/docker”]
}

Ubuntu 25.04

Docker version 29.1.1, build 0aedba5

but I have no zfs errors , it just does not start if the data is on zfs , if I remove daemon.json its works fine creating fresh /var/lib/docker install folder but thats on my boot drive / btrfs

tried with these settings removing graph for the snapshotter

{
“data-root”: “/ZFS3WAY24B/docker”,
“storage-driver”: “zfs”,
“features”:{
“containerd-snapshotter” : true
}

#“graph”: “/ZFS3WAY24B/docker”

}



systemctl status docker.service
× docker.service - Docker Application Container Engine
     Loaded: loaded (/usr/lib/systemd/system/docker.service; enabled; preset: enabled)
    Drop-In: /etc/systemd/system/docker.service.d
             └─override.conf
     Active: failed (Result: exit-code) since Sun 2025-11-30 20:52:32 AEST; 45s ago
   Duration: 19.555s
 Invocation: d084d1437de94bd1929d2aac78b47f3f
TriggeredBy: × docker.socket
       Docs: https://docs.docker.com
    Process: 1715092 ExecStart=/usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock (code=exited, status=1/FAILURE)
   Main PID: 1715092 (code=exited, status=1/FAILURE)

Nov 30 20:52:32 aio systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
Nov 30 20:52:32 aio systemd[1]: docker.service: Start request repeated too quickly.
Nov 30 20:52:32 aio systemd[1]: docker.service: Failed with result 'exit-code'.
Nov 30 20:52:32 aio systemd[1]: Failed to start docker.service - Docker Application Container Engine.

journalctl -xeu docker.service
░░ The job identifier is 12415415 and the job result is failed.
Nov 30 20:52:32 aio systemd[1]: docker.service: Scheduled restart job, restart counter is at 3.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ Automatic restarting of the unit docker.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Nov 30 20:52:32 aio systemd[1]: docker.service: Start request repeated too quickly.
Nov 30 20:52:32 aio systemd[1]: docker.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ The unit docker.service has entered the 'failed' state with result 'exit-code'.
Nov 30 20:52:32 aio systemd[1]: Failed to start docker.service - Docker Application Container Engine.
░░ Subject: A start job for unit docker.service has failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░ 
░░ A start job for unit docker.service has finished with a failure.
░░ 
░░ The job identifier is 12416145 and the job result is failed.

zodac · December 9, 2025, 11:03am

Is it possible to get this working on v29.1.2? getting the following on start-up:

INFO[2025-12-10T00:00:39.909691465+13:00] Starting up
INFO[2025-12-10T00:00:39.910140054+13:00] OTEL tracing is not configured, using no-op tracer provider
INFO[2025-12-10T00:00:39.910201893+13:00] CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory dir=/etc/cdi
INFO[2025-12-10T00:00:39.910214393+13:00] CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory dir=/var/run/cdi
INFO[2025-12-10T00:00:39.919508674+13:00] Creating a containerd client address=/run/containerd/containerd.sock timeout=1m0s
INFO[2025-12-10T00:00:39.924593297+13:00] Loading containers: start.
INFO[2025-12-10T00:00:39.924640166+13:00] [graphdriver] trying configured driver: zfs
failed to start daemon: error initializing graphdriver: prerequisites for driver not satisfied (wrong filesystem?): zfs

When adding the following to /etc/docker/daemon.json:

“features”: {
“containerd-snapshotter”: false
}

I just get a different error:

INFO[2025-12-10T00:01:47.867783275+13:00] Starting up
INFO[2025-12-10T00:01:47.868320874+13:00] OTEL tracing is not configured, using no-op tracer provider
INFO[2025-12-10T00:01:47.868402787+13:00] CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory dir=/etc/cdi
INFO[2025-12-10T00:01:47.868412927+13:00] CDI directory does not exist, skipping: failed to monitor for changes: no such file or directory dir=/var/run/cdi
WARN[2025-12-10T00:01:47.876770206+13:00] “containerd-snapshotter” is now the default and no longer needed to be set
INFO[2025-12-10T00:01:47.877299268+13:00] Creating a containerd client address=/run/containerd/containerd.sock timeout=1m0s
INFO[2025-12-10T00:01:47.882555691+13:00] Loading containers: start.
INFO[2025-12-10T00:01:47.882592654+13:00] Starting daemon with containerd snapshotter integration enabled
WARN[2025-12-10T00:01:47.884579145+13:00] Preferred snapshotter not available in containerd message=“lstat /var/lib/containerd/io.containerd.snapshotter.v1.zfs: no such file or directory: skip plugin”
failed to start daemon: configured driver “zfs” not available: unavailable

Running into issues even when I downgrade to v28 - do I need a clean reinstall at this point?

Topic		Replies	Views
ZFS pool suddenly unmounted by Docker (maybe)? General	13	921	April 9, 2024
Questions about zfs volumes and networks being created General	1	855	June 10, 2018
Docker Daemon stops from time to time General docker , zfs , lxc , proxmox	12	1684	January 16, 2023
Function not implemented error only inside docker container (ZFS) General	1	3935	January 27, 2015
Being naughty with ZFS Testcontainers docker	0	949	September 4, 2017

Related topics