I have an application that uses spring-cloud-consul to register with a consul agent whose address I give as an environment variable to the container. This works fine when deploying on one machine with one consul server.
I am now trying to scale this to work with swarm. My goal is to have one consul agent (in either client or server mode) on every swarm node and have containers that are deployed through swarm connect with their local consul agent. The issue is that because I am deploying through swarm I don’t know what node they will end up on so I can’t tell the container ahead of time. There does not appear to be a way specifying the dockerhost in a compose file because swarm does not know where it is deploying ahead of time and there are multiple open requests such as: https://github.com/docker/swarm/issues/1106. Similarly it does not appear this could be set with environment variables as swarm will use the environment variables on on the node you are deploying from.
The next idea was to add a local bridge network to every node. I would then register the consul agents on both the overlay and the local bridge networks where it would be given a specific ip. The issue again is that swarm does not appear to let me specify that a service should join the local bridge network. I seem to have to give it the nodeid/network name which defeats the purpose as I don’t know where it will be deployed.
Has anyone managed to solve a similar issue when deploying services on swarm that need to find other local service?