Single service with per-container configuration dependent on hostname

General
1

Hi all!

I would like to create a stack that defines 1 service over n containers on n nodes. Here are the parameters:

  1. Most of these containers will need to NFS mount with its host node (via the node’s DNS hostname) but some will mount an ordinary volume instead.
  2. Each volume should have the name “servicename_nodename” (where “nodename” is the node’s DNS hostname, as above).
  3. Additionally, each container should be configured with the hostname of “servicename.nodename”.

I am able to define a lengthy docker-compose.yml file that statically configures this as n services but I would really like to reduce this to 1 generalized, dynamic service. I have fiddled a bit with labels but cannot seem to get what I am after.

Is the configuration I am after even possible?

2

Can’t you use global instead of replicate ?

3

Although it is new to me, it seems that YAML is a sufficiently advanced language that it shouldn’t matter if I do or do not.

4

A service is a wrapper for one or more replicas of the same set of container instances on one or more nodes. It seems like you try to use a service in a way where you want to create and run distinct services with “instance identity” (different config+state per node) purely by the convention on which node they run on.

You can try to leverage template variables to override the name of the actualy created volume and try to use assign a network alias for servicename.nodename. Though, I am not sure if an alias is allowed to have dots.

5

A service is a wrapper for one or more replicas of the same set of container instances on one or more nodes.

I get that.

It seems like you try to use a service in a way where you want to create and run distinct services with “instance identity” (different config+state per node) purely by the convention on which node they run on.

You’ve got it half right. I specifically stated (twice) that I would like to have 1 service, not distinct services. I see the “instance identity” that you describe (which I am indeed after) as being affected per container.

You can try to leverage template variables to override the name of the actualy created volume and try to use assign a network alias for servicename.nodename. Though, I am not sure if an alias is allowed to have dots.

Could you provide an example in YAML or direct me to where I could find more information?

For what it’s worth I am also trying to manage all of this in Portainer.

6

Sure I can. A forum search for “template variables” brings up an old post of mine: Example usage of docker swarm template placeholders

They can only be used on certain elements. I used them on services environment elements and within volume declarations. Neither service name, nor volume names can be templated.

update:

The volume part will high likely look something like this:

version: "3.9"
services
  example-service:
    ...
    volumes:
      - example-volume:/container/path
...
volumes:
  example-volume:
    name: "{{.Service.Name}}_{{.Node.Hostname}}"
    driver_opts:
      type: "nfs"
      o: "addrx.x.x.x,nolock,soft,rw"
      device: ":/share/{{.Service.Name}}/{{.Node.Hostname}}

Means for the code snippet: example-service and example-volume can not be templated. volumes name actualy should be templateable and effects the name of the volume and is used without the stack prefix.

7

Thanks for that, very helpful.

For the sake of clarity, I’m not interested in templating the service name, just the volume names and hostnames inside the containers.

The snippet you supplied would work for the containers that require NFS mounts but not the ones that would require ordinary Docker volumes.

Could I define 2 extension fields such as the following and pull their configs in based on some variable (such as the hostname)?

x-nfs-volume:
  &nfs-volume
  example-volume:
    name: "{{.Service.Name}}_{{.Node.Hostname}}"
    driver_opts:
      type: "nfs"
      o: "addrx.x.x.x,nolock,soft,rw"
      device: ":/share/{{.Service.Name}}/{{.Node.Hostname}}

x-docker-volume:
  &docker-volume
  example-volume:
8

I just wanted to emphasis the limitations of templates.

the key is the “name” element, this is what actualy is responsible for the volume name in the docker engine. This allows to reference a declared volume name inside the compose file, but maps to a different name on the engine level.

x-docker-volume:
  &docker-volume
  example-volume:
    name: "{{.Service.Name}}_{{.Node.Hostname}}"

The approach works for bind-mounts as well (where a host folder is mounted into a container folder). The parameters just look slightly different.

One more thing: once a volume is declared, changes to its configuration in the compose file will not be reflected. They need to be deleted manualy on each node, in order to be re-created with the changed configuration.

Not that I am aware of. The only solution I see is to solve this problem on the host level. Either use local storage or mount nfs folders into the folders you bind-mount into the container. Though, this approaches will suffer when nfs mounts become stale. If nfs backed volumes are used, stale mounts should be recovered by docker itself.

9

I am thinking the thing to do would be to create volumes manually on each node, call them for example “data”, and use “external: true” in the volume configuration. This would obviate the need to have volume names include the hostname of the containers.

I’m not sure if NFS is supported by docker itself or if it is just a feature to be had in docker-compose but that would certainly simplify things. EDIT: Bingo, here it is! https://codeopolis.com/posts/mounting-nfs-shares-as-docker-volumes/

I am aware about the problems with stale NFS mounts but it’s just something I have to deal with for the time being.