Colocate containers in docker swarm

I have two services (serviceA and serviceB). ServiceA is a java application and serviceB is a cache. I would like to always deploy containers from these services together (for example containerA1 on host1 and containerB1 on same host). ContainerA1 should always communicate with the containerB1 (on the same host).

I have tried to define two services in docke-compose and define network aliases for them, but they are not colocated, and even if they are colocated (if I deploy them on all machines) they use round robin to communicate and I would like them to communicate with the container on the same host.

version: '3.1'
services:
  serviceA:
    ...
    deploy:
      replicas: 3
    network:
      testnetwork:
        aliases: ["serviceA"]

    serviceB:
      ...
      deploy:
        replicas: 3
      network:
        testnetwork:
          aliases: ["serviceB"]

networks:
  testnetwork:
    driver: overlay

I would like the containers to be colocated and to communicate with the containers on the same host, like in the diagram.

docker-swarm

2 Likes

If this is a caching thing that you are looking at, why not have the cache in the same container as the service using it. Since you want a 1 <-> 1 relationship there is no real reason to have 2 separate containers is there?

On a a separate note (and I am sure you have thought about it) I would think that you would want the cache to be available across the various containers otherwise you are caching multiple times for the same thing.

The local cache will need to be accessed by more than one container on the same host, for example I will have a third service (serviceC) that want to access to local cache as well. Another reason to keep the cache in a separate container is to isolate the app process from the cache ones as they are developed by different teams.

I want the cache data to be replicated in each container to improve performance at the cost of memory (as the same data is replicated on each host).

I think the only way you’re going to be able to both have a dedicated cache per host as well as being “scaled” across the swarm cluster is to duplicate the services (consumers and cache) in your config and then use placement constraints to ensure that your duplicated services are spread across the swarm nodes. Something like:

version: '3.1'
services:
  ConsumerA:
    ...
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeA'
    network:
      testnetwork:
        aliases: ["ConsumerA"]

    CacheA:
      ...
      deploy:
        replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeA'
      network:
        testnetwork:
          aliases: ["CacheA"]

  ConsumerB:
    ...
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeB'
    network:
      testnetwork:
        aliases: ["ConsumerB"]

    CacheB:
      ...
      deploy:
        replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeB'
      network:
        testnetwork:
          aliases: ["CacheB"]

  ConsumerC:
    ...
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeC'
    network:
      testnetwork:
        aliases: ["ConsumerC"]

    CacheC:
      ...
      deploy:
        replicas: 1
      placement:
        constraints:
          - node.hostname == 'nodeC'
      network:
        testnetwork:
          aliases: ["CacheC"]

networks:
  testnetwork:
    driver: overlay

You can use other constraints (such as node.labels) in place of node.hostname, but you get the point.

There are issues that could arise in this situation (such as when a node goes down and the constraints cannot be satisfied), but it is a way of getting your desired node-local caching.

In Kubernetes you have the concept of Pods which can house multiple containers, where the containers in a single Pod instance all run on the same host (and communicate over a loopback interface). In that scenario, you can replicate Pods across the cluster which would achieve your desired configuration. In general though, Kubernetes has a much greater learning curve in terms of setting up a cluster and configuring/deploying your application containers.

HTH

@annchurch thank you for the suggestion, but I would like the cluster to scale automatically.

At the moment the best solution I have found is to create a daemon on each host. The daemon will take containerB’s IP and write it to a file, containerA will mount the file and will read the ip from it.