Docker Swarm NFS mount

Open Source Projects Swarm
,
1

Docker Swarm NFS mount

Hi all,

I have spent now two day googling and trying to mount NFS volumes via my “compose” file but to no avail. The closest I got is container hanging with:

ID                  NAME                                            IMAGE               NODE                DESIRED STATE       CURRENT STATE           ERROR               PORTS
z7q6ujnaemqs        jenkins_master_jenkins-master.1   jenkins:latest                          Running             Pending 5 minutes ago

And this is the “compose” file:

version: "3.2"
services:
  jenkins-master:
    image: jenkins
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "1"
          memory: 1G
      restart_policy:
        condition: on-failure
    ports:
      - "8080:8080"
    volumes:
          - target: /opt/jenkins_home
            type: volume
            volume_driver: local
            volume_opts:
              type: nfs
              device: addr=[NFS/EFS SERVER'S HOSTNAME]:/
              o: nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
    networks:
      - jenkins-master

Server Version: 17.06.2-ce

Is this even possible? I assumed that it has to be but I’m rather doubtful now…

Please help!

2

I have successfully used the ContainX/docker-volume-netshare. To not answer your question directly. :slight_smile:

There are others as well.

Using the local driver this post might help.
Without having it tested myself, they suggest this compose file snippet:

    volumes:
      - type: volume
        source: example
        target: /nfs
        volume:
          nocopy: true
volumes:
  example:
    driver_opts:
      type: "nfs"
      o: "addr=10.40.0.199,nolock,soft,rw"
      device: ":/docker/example"

Which looks slightly different then your device and option section.

3

Hi Christian,

I tried this as well but no luck - it kept hanging. Also the plugin set-up didn’t seem to be too convenient when it comes to swarm setup. In the end I decided to go with “docker service create …” form as this works. Will give the “copose/swarm” another try in some time.

Thanks for you input!

Regards

4

Hi.

We have gotten it working with a stack file (aka compose v3+).

Unfortunately I do not have a sample handy, because we went a different route (will explain that below).
One thing I do remember about the compose option is that you MUST have the NFS client install on all swarm hosts your container might run on.

Ok, so after I discovered that docker will not insulate you from the requirement that the storage client (NFS in this case) MUST be installed on the host in ANYWAY, we decided to handle the NFS mount on the host level.
Fairly easy to achieve with tools such as ansible, chef, puppet or whatever.

You could also create a “golden VM” that has the mount config baked in, and just start all your hosts from that.

The container mount then is just a simple, normal “external file system” mount.

For example:
my-server:
image: someimage
volumes:
- /data/myapp/:/myapp/data

As a general side note: if you are unhappy with persistent storage options in swarm setups at the moment, believe me, you are NOT alone.
Basically it all sorta sucks, although I hear good things about rexray. BUT, again, host-level intervention is required afaik.

5

Hi Hendrik,

As I matter of fact I got it working too - last night I decided to give it another try and TA-DA! It worked! Looks like I was fairly close to it before. You are right though, you do need the NFS client on the host installed.

I tried also with mounting it on the host but that works only when one container is trying to then use this mount as a volume with every next one failed to do so.

Below is my docker-compose.yml as an example:

version: "3.2"
services:
  jenkins-master:
    image: jenkins
    deploy:
      replicas: 1
      resources:
        limits:
          cpus: "1"
          memory: 1G
      restart_policy:
        condition: on-failure
    ports:
      - "8080:8080"
    volumes:
      - "jenkins_home:/var/jenkins_home/"
      - "/var/run/docker.sock:/var/run/docker.sock"
    networks:
      - infrastructure

volumes:
  jenkins_home:
    driver: local
    driver_opts:
      type: nfs
      o: addr=fs-c000000c.efs.eu-west-1.amazonaws.com,nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2
      device: fs-c00000c.efs.eu-west-1.amazonaws.com:/docker/volumes/jenkins_home/

networks:
  infrastructure:

Just replace relevant pieces with your details and enjoy the persistence :wink:

1 Like
6

I am thinking of adding NFS mounts to my “farm” right now I am using a SMB mount since the host OS is Windows. However, I am missing some key things that some container images such as GitLab require specifically permissions.

My question about using NFS is primarily with regards to concurrency and write conflicts on a larger scale. I am also thinking that the NFS mounts will be proper VMs just configured to be NFS stores not docker containers.

7

BTW I recommend you use the TLS connection from Jenkins to docker if you plan to use the docker-slaves plugin rather than giving full control of the socket.