Unable to use a NFS-mounted directory to share volumes between Docker engines

General Discussions
1

Hi there,

I’d like to create a shared folder between 2 docker hosts. This shared folder uses NFS v4 and the exported share is mounted on /var/lib/docker/volumes on each docker host.

This configuration worked well with Docker 1.10.x but is not working anymore with Docker 1.11.x (tried 1.11.0 / 1.11.1).

My setup is (virtualbox machines):

  • Ubuntu 14.04 node, the NFS server
  • Ubuntu 14.04, docker engine #1 (docker version 1.11.1)
  • Ubuntu 14.04, docker engine #2 (docker version 1.11.1)

I am using the default storage driver (aufs) and I am able to start the first docker engine #1 without issues. However, the docker engine #2 won’t start.

I’ve noticed the existence of the following file after starting the first docker engine: /var/lib/docker/volumes/metadata.db. I never saw this file using Docker 1.10.x, was it added in 1.11.x?

If I delete this file, the second docker engine #2 will be able to start. I don’t think this is recommended though.

More information about my setup (you can reproduce it using virtualbox with this vagrant setup: https://github.com/deviantony/vagrant-swarm-cluster/tree/nfs)

Content of the /etc/exports file on the NFS server:

/var/nfs IP-DOCKER-ENGINE-1(rw,sync,no_subtree_check,no_root_squash)
/var/nfs IP-DOCKER-ENGINE-2(rw,sync,no_subtree_check,no_root_squash)

Docker version:

$ docker version
Client:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.1
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   5604cbe
 Built:        Tue Apr 26 23:30:23 2016
 OS/Arch:      linux/amd64

Docker info:

docker info
Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 1
Server Version: 1.11.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 10
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: null host overlay bridge
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 489.1 MiB
Name: swarm-manager
ID: DKFU:OG4R:POZA:HCGM:Q65H:OG6P:7FDW:ISBG:BVYB:CMVT:NKQV:V4MQ
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
Cluster store: consul://CONSUL-IP:8500
Cluster advertise: NODE-IP:2375
2

I wouldn’t recommend trying to share volumes between engines like this at all.

There are some implementations of a docker volume driver that uses an nfs share as its source. Take a look at https://github.com/SvenDowideit/docker-volumes-nfs and https://github.com/gondor/docker-volume-netshare

3

Thanks for the answer.

For those interested, I managed to create a similar setup using the following plugin: https://github.com/CWSpear/local-persist

You can retrieve the setup (Swarm cluster in Vagrant) here: https://github.com/deviantony/vagrant-swarm-cluster/tree/nfs-local-persist