I have a problem with services which used an NFS server as volume bind. The compose file had a section like this:
xwiki-data:
driver: local
driver_opts:
type: "nfs4"
o: addr=123.45.67.89,nolock,soft,rw
device: ":/raid/nfs/swarm/ecap-wiki/xwiki"
but I have issues with the NFS server and wanted switch to a local bind (GlusterFS which is mirrored to all nodes). So I changed the volume section to:
xwiki-data:
driver: local
driver_opts:
type: none
o: bind
device: "/data/glusterfs/swarm/ecap-wiki/xwiki"
copied over all the data with rsync -avz ...
and then did a
docker stack rm SERVICENAME
docker deploy -c config.yml SERVICENAME
The service is now constantly crashing due to a “connection refused”, because it is trying to access the old NFS share. I don’t understand what’s going on. I also deleted the Docker volume and redeployed a few times but the error always says that it’s still accessing the NFS:
# docker service ps --no-trunc ecap-wiki_web
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
yz7s0dr6qtyzk312o7wm9ehft ecap-wiki_web.1 xwiki:lts-mysql-tomcat@sha256:91b56f7635dc9031cfc777f8bbccc955b3c38b575cc1c30e27b19c802441ffe7 pi1080 Running Starting less than a second ago
6xowo1k9meyflfilp6bym7ft4 \_ ecap-wiki_web.1 xwiki:lts-mysql-tomcat@sha256:91b56f7635dc9031cfc777f8bbccc955b3c38b575cc1c30e27b19c802441ffe7 pi1085 Shutdown Rejected 10 seconds ago "failed to mount local volume: mount :/raid/nfs/swarm/ecap-wiki/xwiki:/var/lib/docker/volumes/ecap-wiki_xwiki-data/_data, data: addr=123.45.67.89,nolock,soft: connection refused"
vbh3hs3uf02py1syvnf8joj2h \_ ecap-wiki_web.1 xwiki:lts-mysql-tomcat@sha256:91b56f7635dc9031cfc777f8bbccc955b3c38b575cc1c30e27b19c802441ffe7 pi1085 Shutdown Rejected 15 seconds ago "failed to mount local volume: mount :/raid/nfs/swarm/ecap-wiki/xwiki:/var/lib/docker/volumes/ecap-wiki_xwiki-data/_data, data: addr=123.45.67.89,nolock,soft: connection refused"
tpjdx584q63i7qeoj40d8m2au \_ ecap-wiki_web.1 xwiki:lts-mysql-tomcat@sha256:91b56f7635dc9031cfc777f8bbccc955b3c38b575cc1c30e27b19c802441ffe7 pi1085 Shutdown Rejected 20 seconds ago "failed to mount local volume: mount :/raid/nfs/swarm/ecap-wiki/xwiki:/var/lib/docker/volumes/ecap-wiki_xwiki-data/_data, data: addr=123.45.67.89,nolock,soft: connection refused"
p9iy3zr13hiut0ub2mnginl0t \_ ecap-wiki_web.1 xwiki:lts-mysql-tomcat@sha256:91b56f7635dc9031cfc777f8bbccc955b3c38b575cc1c30e27b19c802441ffe7 pi1085 Shutdown Rejected 25 seconds ago "failed to mount local volume: mount :/raid/nfs/swarm/ecap-wiki/xwiki:/var/lib/docker/volumes/ecap-wiki_xwiki-data/_data, data: addr=123.45.67.89,nolock,soft: connection refused"
Any ideas why where this configuration might be cached, or what’s going on here? I did this docker stack rm ...
and docker deploy ...
combination a lot of time in past when changing the service configuration and it always worked. I thought that docker stack rm
would remove everything…
EDIT: Btw. the only workaround so far is to choose a different name when deploying the stack, but this is of course not a solution.