Docker and NFS Share

Hello,

I am trying to use containers in order to build a personal cloud. I am using a NFS server and my VM host is a NFS client, i am trying to use this NFS share in my containers.

Concrete use case :
Seafile Deployment using this example : https://download.seafile.com/d/320e8adf90fa43ad8fee/files/?p=/docker/docker-compose.yml

After many times reading docs, it seems that i have to use docker volume, so i build few volumes like this :

docker volume create --driver local --opt type=nfs --opt o=addr=192.168.2.13,rw --opt device=:/volume1/Cloud/Seafile/MYSQL seafile_mysql

root@Seafile:~# docker volume inspect seafile_data 
[
    {
        "CreatedAt": "2019-10-11T21:30:59+02:00",
        "Driver": "local",
        "Labels": {},
        "Mountpoint": "/var/lib/docker/volumes/seafile_data/_data",
        "Name": "seafile_data",
        "Options": {
            "device": ":/volume1/Cloud/Seafile/DATA",
            "o": "addr=192.168.2.13,rw",
            "type": "nfs"
        },
        "Scope": "local"
    }
]

I use it in the yml file, i run the containers, i was happy, i put few files in the containers that is using my NFS share and i see that the empty space on host VM was reducing ! which is not expected as it should be stored on the remote NFS server.

root@Seafile:~# df -h
Filesystem                   Size  Used Avail Use% Mounted on
udev                         983M     0  983M   0% /dev
tmpfs                        200M   11M  189M   6% /run
/dev/sda1                     14G  4.1G  9.0G  32% /
tmpfs                        998M     0  998M   0% /dev/shm
tmpfs                        5.0M     0  5.0M   0% /run/lock
tmpfs                        998M     0  998M   0% /sys/fs/cgroup
tmpfs                        200M     0  200M   0% /run/user/1000
192.168.2.13:/volume1/Cloud  2.7T  2.2T  549G  81% /mnt/Seafile
//192.168.2.13/Cloud         2.7T  2.2T  549G  81% /mnt/cifs
overlay                       14G  4.1G  9.0G  32% /var/lib/docker/overlay2/6b19ddd89720c38c2fe2bd0a0932ae5d8716961282dcec7828459078755b8ac6/merged
overlay                       14G  4.1G  9.0G  32% /var/lib/docker/overlay2/6f5fff28267145804887bc6c0944fce8ad53af8988427dba8bf155c2644fbdbf/merged
shm                           64M     0   64M   0% /var/lib/docker/containers/96821b1029af2a826e7ab8e42da1ca3fda2c5d2ea64c461b4ced3d71fba1c738/mounts/shm
shm                           64M     0   64M   0% /var/lib/docker/containers/776cc8604d6fedae3d9ebbd5ed8d84e48faaa7ea99bdec7d9b48ac4ed613565f/mounts/shm
overlay                       14G  4.1G  9.0G  32% /var/lib/docker/overlay2/405c1abe0cd00b63a5f6ed2c7b2f906ddd0cf42dcead1568296e9c82f97b62b6/merged
shm                           64M     0   64M   0% /var/lib/docker/containers/b0b778ca69bcff230477d81f4f84af07596149c16e8ae1acf0ad950445d4f7ba/mounts/shm

Can someone tell me what i am doing wrong ?
I have no idea for now, if needed i put my whole yml file to give you all inputs, so sorry for the big post :slight_smile:

root@Seafile:~# cat docker-compose.yml 
version: '2.0'
services:
  db:
    image: mariadb:10.1
    container_name: seafile-mysql
    environment:
      - MYSQL_ROOT_PASSWORD=mypassword  # Requested, set the root's password of MySQL service.
      - MYSQL_LOG_CONSOLE=true
    volumes:
      - seafile_mysql:/var/lib/mysql  # Requested, specifies the path to MySQL data persistent store.
    networks:
      - seafile-net

  memcached:
    image: memcached:1.5.6
    container_name: seafile-memcached
    entrypoint: memcached -m 256
    networks:
      - seafile-net
          
  seafile:
    image: seafileltd/seafile-mc:latest
    container_name: seafile
    ports:
      - "80:80"
    volumes:
      - seafile_data:/shared   # Requested, specifies the path to Seafile data persistent store.
    environment:
      - DB_HOST=db
      - DB_ROOT_PASSWD=mypassword  # Requested, the value shuold be root's password of MySQL service.
      - TIME_ZONE=Europe/  # Optional, default is UTC. Should be uncomment and set to your local time zone.
      - SEAFILE_ADMIN_EMAIL=mail@mail.com # Specifies Seafile admin user, default is 'me@example.com'.
      - SEAFILE_ADMIN_PASSWORD=mypassword     # Specifies Seafile admin password, default is 'asecret'.
      - SEAFILE_SERVER_LETSENCRYPT=false   # Whether to use https or not.
      - SEAFILE_SERVER_HOSTNAME=seafile.myserver.local # Specifies your host name if https is enabled.
    depends_on:
      - db
      - memcached
    networks:
      - seafile-net

networks:
  seafile-net:

volumes:
  seafile_data:
  seafile_mysql:

Thanks !

If you want an external created volume to be used within a compose.yml, you need to declare it as “external: true”, e.g.:

volumes:
  seafile_data:
    external: true
  seafile_mysql:
    external: true

Though, instead I would advise to declare the nfs volume directly inside the compose.yml:

volumes:
  seafile_data:
    driver_opts:
      type: nfs 
      o: addr=192.168.2.13
      device: :/volume1/Cloud/Seafile/DATA
  seafile_mysql:
    driver_opts:
      type: nfs 
      o: addr=192.168.2.13
      device: :/volume1/Cloud/Seafile/MYSQL

Though, volume declarations are created on first start and are immutable, if you want changes on the driver_opts to be used, you will need to remove the volume manualy (docker volume rm) and restart the compose stack to re-register the volume with changed options.

Thanks Meyay for your complete answer and your examples.
I tried both but i still have a permission issue.

To illustrate that :

If i use local folder within my yml file it’s all good.

When using the external method or by creating directly the volume within the compose file (which is the best way i agree with you) i get the same result which is :

seafile | *** Running /etc/my_init.d/01_create_data_links.sh...
seafile | mv: failed to preserve ownership for ‘/shared/logs/var-log/apt/eipp.log.xz’: Operation not permitted
seafile | mv: failed to preserve ownership for ‘/shared/logs/var-log/apt/history.log’: Operation not permitted
seafile | mv: failed to preserve ownership for ‘/shared/logs/var-log/apt/term.log’: Operation not permitted
seafile | mv: failed to preserve ownership for ‘/shared/logs/var-log’: Operation not permitted
seafile | *** Running /etc/my_init.d/10_syslog-ng.init…
seafile | /bin/chown: changing ownership of ‘/var/log’: Operation not permitted
seafile | *** /etc/my_init.d/10_syslog-ng.init failed with status 1
seafile |
seafile | *** Killing all processes…

All the users of my systems have read/write access, this NFS share is “all users mapped to admin” :

root@Seafile:~# touch /mnt/Seafile/Seafile/DATA/TEST_ROOT
nuc@Seafile:~$ touch /mnt/Seafile/Seafile/DATA/TESTNUCUSER
nuc@Seafile:~$ ls -la /mnt/Seafile/Seafile/DATA/
total 16
drwxrwxrwx 3 1024 users 4096 Oct 12 21:55 .
drwxrwxrwx 3 1024 users 4096 Oct 12 21:51 …
drwxrwxrwx 4 1024 users 4096 Oct 12 21:51 logs
-rwxrwxrwx 1 1024 users 0 Oct 12 21:55 TESTNUCUSER
-rwxrwxrwx 1 1024 users 0 Oct 12 21:55 TEST_ROOT

I don’t understand why containers are not able to work with this NFS share…
Thanks

It is not only related to docker… it’s related to my Synology NFS Server i guess.
I reproduce the issue with the following workflow :

root@Seafile:~# touch test
root@Seafile:~# chmod 640 test 
root@Seafile:~# cp test /mnt/Seafile/
root@Seafile:~# ls -la /mnt/Seafile/
total 24
drwxrwxrwx 5 nuc  nogroup 4096 Oct 12 22:04 .
drwxr-xr-x 4 root root    4096 Oct 10 00:23 ..
drwxrwxrwx 3 1024 users   4096 Oct 12 22:00 data
drwxrwxrwx 3 1024 users   4096 Oct 12 21:51 Seafile
-rw-r----- 1 1024 users      0 Oct 12 22:04 test
root@Seafile:~# mv test /mnt/Seafile/test2
mv: failed to preserve ownership for '/mnt/Seafile/test2': Operation not permitted

It’s corrected ! For people who might have the same setup/issue and Googled that :
You MUST NOT check the box “Map the users to admin” as it will destroy the ownership of the file and will cause issue (as described above)
Thanks to that it is now solved and again thanks Meyay ! I googled so long before having all the info that you shared and even after googling i was not aware of your first method (external volume).

Next step : Try to see if NFS v4 is supported by docker :slight_smile:

Edit : Here is the command that is used by docker for NFS v4 :slight_smile:
docker volume create --driver local --opt type=nfs4 --opt o=addr=192.168.2.13,rw --opt o=proto=tcp --opt o=port=2049 --opt device=:/volume1/Cloud nfs4

Some arguments are not supported in composed yaml file even if they are in the docker CLI :

    volumes:
  seafile_data_nfs4:
    driver_opts:
      type: nfs4
      o: addr=192.168.2.13
#      o: proto=tcp
#      o: port=2049
      device: :/volume1/Cloud/Seafile/DATA
  seafile_mysql_nfs4:
    driver_opts:
      type: nfs4
      o: addr=192.168.2.13
#      o: proto=tcp
#      o: port=2049
      device: :/volume1/Cloud/Seafile/MYSQL

If i uncomment port and proto it’s doesn’t work anymore.

I am mounting my NFSv4 shares in my homelab like this:

driver_opts:
  type: nfs 
  o: addr=192.168.2.13,nfsvers=4
  device: :/volume1/Cloud/Seafile/DATA

Yaml is basicly a key/value map. A key declared more than once will either lead to a duplicate key error or one of the declaration will be used. You can not have more than one o: key.