Swarm + NFS Volume = Missing mount options

Docker Engine Swarm
,
1

Hey everyone,

I am running into an issue with Docker Swarm in combination with NFS-mounted volumes, in particular when I “migrate” a server to another node in the swarm.

Some context: I have a docker swarm with multiple nodes. I have one node I use as test environment and multiple others which I use as production environment. I use placement constraints to tell the services where to go.

I recently was playing around with Ntfy and decided to use it for UptimeKuma notifications. Once testing was done I moved it to my production node (1 replica). The service started fine, but the configuration seemed to be off. I checked the container’s data directory and found it was empty. I then checked the host and found that the volume, even though created, was not mounted.

When I did an inspect of the volumes of the hosts of my test- and production environment, I noticed that on my production environment the mount options were missing:

Volume inspect of test node (this is also the node on which I initially created the volume using portainer):

docker volume inspect Ntfy
[
    {
        "CreatedAt": "2023-02-17T14:41:36+01:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/Ntfy/_data",
        "Name": "Ntfy",
        "Options": {
            "device": ":/export/xxxxxxxx/Ntfy",
            "o": "addr=xxxxxxxxxx,rw,noatime,rsize=8192,wsize=8192,tcp,timeo=14,nfsvers=4",
            "type": "nfs"
        },
        "Scope": "local"
    }
]

Inspect of the volume created by docker swarm on the production environment after I migrated the service:

docker volume inspect Ntfy
[
    {
        "CreatedAt": "2023-02-22T17:07:31+01:00",
        "Driver": "local",
        "Labels": null,
        "Mountpoint": "/var/lib/docker/volumes/Ntfy/_data",
        "Name": "Ntfy",
        "Options": null, <<<=== ????
        "Scope": "local"
    }
]

Anyone have any idea why the mount options are not passed onto the volume? I suspect this is the reason the volume is not mounted.

2

The output of volume inspect looks like a default named volume (=without any parameters) was created. Please share how you created the volume, and details like docker node ls from a manager node, and docker info + docker version of the node where it doesn’t work.

Notes about volumes:

  • local driver mean that the volume is managed locally on a node, regardless whether it is a remote share or not. If a volume needs to be deleted, it must be done individually on each node the volume exists!
  • stack deployments: a volume is created on a node, the first time a container using it deployed on it.
  • stack deployments: a volume declaration is immutable, changes in a compose file are not reflected back to the configuration. To apply changes, the existing volume declaration must be deleted and re-created with the new parameters.
3

@meyay

The volume with options was created through the Portainer UI. The one with the options missing was created by docker swarm after I moved the service to the other node.

Docker node ls (redacted):

HOSTNAME                 	STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
mngr.local.com             	Ready     Active         Leader           20.10.20
dev.local.com    			Ready     Active                          20.10.20
ext.local.com   			Ready     Active                          20.10.20
int.local.com    			Ready     Active                          20.10.20
play.local.com    			Ready     Active                          20.10.20

Docker info of the failing node (redacted):

Client:
 Context:    default
 Debug Mode: false

Server:
 Containers: 2
  Running: 2
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 20.10.20
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: 2nnm268n9r366yg7naq4ug0pl
  Is Manager: false
  Node Address: 192.x.x.16
  Manager Addresses:
   192.x.x.12:2377
 Runtimes: io.containerd.runtime.v1.linux runc io.containerd.runc.v2
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 9cd3357b7fd7218e4aec3eae239db1f68a5a6ec6
 runc version: 5fd4c4d144137e991c4acebb2146ab1483a97925
 init version:
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.15.93-0-lts
 Operating System: Alpine Linux v3.16
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.745GiB
 Name: ext.local.com
 ID: ZBZP:USAG:Z3MJ:SFNH:F3C7:SSAB:2CZB:5PT6:JV2Y:LWS4:X63P:HANA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

I don’t use stacks at the moment. I just create a volume, then a service and connect the volume to it.

4

Thank you for sharing the details.

I guess this is what causes the problem in your situation:

  • a service does not know about the volume declaration, it only knows “I need a volume called x”.
  • a volume is local scoped

With stack deployments the information about the services, volumes and networks exist in the stack state, which is used to create the required volumes using the configuration found in the stack.

You might want to test it with a stack deployment, to identify whether my assumption is correct, or you just experience a bug.

5

@meyay

I thought a stack was just a group of multiple services which, in most cases, communicate with each other. I will try deploying and migrating with a stack to see if that fixes the problem, but I really do not want to create docker compose files for all my services, that is why I am using portainer, so I have a GUI.

In either case, I believe this is a bug. The containers are migrated with all the properties and settings, so should the volumes, otherwise what is the point of having a swarm that can auto-failover to another node if the volume is not spinning up correctly?

6

Indeed, a single swarm stack ideally only includes service that are belonging to a single application or solution.

I should have been more precise earlier: test with a single service that requires a shared volume.

If the problem occurs with stack deployments as well => it’s a bug.

When it comes to swarm services: the scheduler takes care to spread replicas across the nodes because they are global scoped. But if you create a volume using the local driver, I am afraid you get what it says: a locally managed volume. You might want to check if a volume plugin exists that is global scoped (swarm scoped) and supports nfs. For instance, the portworx plugin I was using was global scoped as well: if deleted on one node, it was gone on all of them.

The only thing I can see in your scenario is either to find a global scoped volume plugin that supports nfs, or create the volumes on each node. Though, personally I would do neither of those things and use stacks instead.

7

I just tried it with a stack and the migration worked as it should.

Honestly I’m a bit bummed out that I have to start writing stacks for all my services now, but oh well I guess, I’ll only need to do it once.