Multiple NFS mounts on docker compose

[satrapes]

I have a compose file that has 2 services. I would like to mount 3 directories from the same NFS backend. To do this I will put in a docker-compose.yml file

volumes:
  vol1:
    driver_opts:
      type: "nfs"
      o: "addr=X.Y.Z.W,nolock,soft,rw"
      device: ":/dir/in/nfs/backend1"
  vol2:
    driver_opts:
      type: "nfs"
      o: "addr=X.Y.Z.W,nolock,soft,rw"
      device: ":/dir/in/nfs/backend2"
  vol3:
    driver_opts:
      type: "nfs"
      o: "addr=X.Y.Z.W,nolock,soft,rw"
      device: ":/dir/in/nfs/backend3"
services:
  serv1:
    ...
    volumes:
      - type: volume
        source: vol1
        target: /docker/container/in/dir1
        volume:
          nocopy: true
      - type: volume
        source: vol2
        target: /docker/container/in/dir2
        volume:
          nocopy: true
      ...
    serv2:
      ...
      - type: volume
        source: vol3
        target: /docker/container/in/dir3
        volume:
          nocopy: true
      ...

Now this doesn’t work if there are more than 1 declarations of:

      - type: volume
        source: vol
        target: /docker/container/in/dir
        volume:
          nocopy: true

docker logs will start throwing a lot of messages
rsync: [receiver] mkstemp "/docker/container/in/dir3/file_names" failed: Permission denied
rsync error: some files/attrs were not transferred (see previous errors) code 23) at main.c(1333) [sender=3.2.3]
How can I fix this? One option that I have thought is to actually mount the NFS directories on the host and them bind mount them to the docker containers. Are there any other ways?

[meyay (Metin Y.)]

It should work, if:

• your docker engine runs on linux (I never tried it on Windows/Mac, so can’t realy say wether it works there as well)
• the client is allowed to access the nfs export
• the uid/gid of the process inside the container alligns with the uid/gid of the nfs share

From my experience nfsv4 (or higher) is more reliable/less of a headache than nfvs3. Also are you sure you want to use the nolock option?

I usualy declare my volumes like this:

  data:
    driver_opts:
      type: nfs 
      o: addr=x.x.x.x,nfsvers=4
      device: :/nfs_export/docker/servicename

Works like a charme. Though, I can see why adding soft to use a soft mount make sense.

Update: I had to lookup why I don’t use soft mounts. see: https://linoxide.com/difference-soft-hard-nfs-mount/

[satrapes]

Thanks, I only did that because I found it on this link here on how to setup nfs mount
https://phoenixnap.com/kb/nfs-docker-volumesone
You propose the following right?

volumes:
    data:
        driver_opts:
            type: nfs 
            o: addr=x.x.x.x,nfsvers=4
            device: :/nfs_export/docker/servicename

my_service:
    volumes:
        - data:/dir/in/container

[meyay (Metin Y.)]

You completed the required missing parts of the compose file declaration

[satrapes]

I am afraid that your solution does not seem to work out of the box. It could be something related to the service I am trying to bring up (nextcloud). Out of curiousity have you confirmed that you can have multiple volumes as specified above

[meyay (Metin Y.)]

I use it exactly like that in my homelab with nfsv4 from a synology nas. It doesn’t matter if different exports or the same exports with different subfolders is used: it always works for me.

You might want to check the nfs server configurations, as something seems to be off.

On my docker nodes, the packages nfs-kernel-server and nfs-common are installed.

[satrapes]

Pastebin Online - AppDevTools

I have put in my docker compose file for you to see if you can try it yourself or if you can spot anything obvious. The folders in /mnt/sat_pool/selfhosted_app_data/nextcloud are all owned by www-data:wheel or 33:0 I think. NFS v4 is enabled in TrueNAS. No advanced options have been selected in the NFS service (mapuser, mapgroup, mapall etc)

[meyay (Metin Y.)]

observations:

• version ‘3.4’ is used even though there is zero benefit compared to 2.4
• more volumes declared than used, actualy only 2 nfs baked volumes used
• db service has key command: twice
• values in the volumes section are quoted while unquoted in the services declarations ← inconsistence
• no tags provided.

Personal preferences:

• I would skip driver: local for volumes as its the default driver
• I would use key/value based environment declarations, instead of a yaml sequence
• I would set a specific tag for the images ← no suprise breakage caused by major updates

I am surprised the compose file declaration config you shared actualy works like this. I would have assumed that duplicate keys raise an error and prevent execution.

Note: a volume once created by docker-compose is immutable. Configuration changes will NOT be applied to the volume declaration. In order to update the volume declaration, stop the containers, remove the volumes you need updated and redeploy your compose configuration to let them be re-created with the new values.

update:
I modified your compose file, removed some things irrelevant to me and added others:

version: '2.4'

volumes:
  nextcloud:
    driver_opts:
      type: nfs 
      o: addr=random-ip,nfsvers=4
      device: :/volume3/docker/nextcloud/data
  db:
    driver_opts:
      type: nfs 
      o: addr=random-ip,nfsvers=4
      device: :/volume3/docker/nextcloud/db

services:
  db:
    image: mariadb:10.7
    restart: always
    command: --transaction-isolation=READ-COMMITTED --binlog-format=ROW
    volumes:
      - db:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: password
      MYSQL_DATABASE: nextcloud
      MYSQL_USER: nextcloud
      MYSQL_PASSWORD: password

  app:
    image: nextcloud:23.0-apache
    container_name: nextcloud_app
    restart: always
    ports:
      - 18082:80
    links:
      - db
    volumes:
      - nextcloud:/var/www/html
    environment:
      NEXTCLOUD_TRUSTED_DOMAINS: ${NEXTCLOUD_TRUSTED_DOMAINS}
      NEXTCLOUD_ADMIN_USER: nextcloud
      NEXTCLOUD_ADMIN_PASSWORD: nextcloud
      MYSQL_HOST: db
      MYSQL_DATABASE: nextcloud
      MYSQL_USER: nextcloud
      MYSQL_PASSWORD: password

I had to remove the volumes you added underneath /var/www/html, it indeed caused a permissions issue (which doesn’t realy make sense to me).

Replace ${NEXTCLOUD_TRUSTED_DOMAINS} in the compose file above either with the hostname of your dockerhost, or with the full qualified domain name you want to use to access your nc instance. By adding these three adition parameters the setup will be skipped all along and you can directly login into nc. There is should be no need to chown the nfs exports, as both images start as root, fix ownership and then start the main process with dropped priviliges.

Update: appearently the subfolder mount stuff realy breaks things. The odd part is that the folders inside the nfs share are correct, but it still doesn’t work. I consider this rather a problem with the nextcloud image itself, than realy a docker problem. You might want to try your luck in the nextcloud forum, I assume the number of nextcloud forum users that actualy run it in docker are magnitudes higher than docker users that actualy run nextcloud. I am personaly not using nextcloud and can’t tell you why it missbehaves like it does.

[satrapes]

You are a legend, thank you very much I will try your changes.