All our compose stacks define 2 networks, 1 shared network - reverse_proxy (that is used to put the outside facing services on, for traefik reverse proxying) and then the internal network for the stack itself (interconnect).
This morning I’ve had a ticket come in that there were problems with the API of one of our stacks.
After long searching, it turned out that the frontend got as IP for “api” the IP of a different stack that had it’s api service connected to the “reverse_proxy” proxy network, instead of the one from the “internal service”.
Is there any way to indicate what the priority is for dns resolving? The order in the compose file obviously does not seem to matter. Or perhaps the priority of the networks?
The simplest would obviously be to demand those connected to the reverse_proxy not to use a generic name but instead use [stackname]_[service} or something, but I’m not a big fan of that. hostname doesn’t work as it’s just an extra added to the dnsnames.
The problem you describe can only happen, if an api service is attached to the reverse-proxy network, so that name resolution sometimes returns the api service from the reverse-proxy network, and sometimes from the internal network.
If I remember right, the frontend can reach its backend in the same compose project using <service name>.<network name>, so in your case api.internal (I don’t recall if this only works for swarm services)
If the service name is not unique amongst all compose projects that use the same network, you need to introduce a network alias for the service in this specific network:
Thus, wort cast: add a network alias to the api service in the internal network, and make the frontend call the api service using the alias.
I didn’t explicitly mention it, but there are indeed multiple APIs connected to that network and that is intended. The reverse_proxy network is where everything that has an outward facing port (and dns) connects to. Since we have multiple APIs facing outwards, more than one of those services is connected to this network.
I was hoping that the api.<network-suffix>. would work, because that would be awesome, but unfortunately api.internal on its own does not resolve so that must probably be swarm only as you suspected.
I did try a network alias, since would be visually more pleasing than renaming the service, except for the unfortunate fact that DNS-wise, this results in an alias, not a rename. It simply adds an extra hostname to that network, on top of the “api” one. Meaning that “api” will still be a known service within the reverse_proxy network.
I did not think about aliasing the API within the internal network though
That would indeed be a much cleaner solution, but that would mean modifying nearly all our stacks instead of just the ones exposing an API to the public.
I guess I’ll ask the rest of the team what they think about it. At this point I don’t really see other alternatives either.
For future me/other people, this does work and looks fairly clean:
root@e9d0d67d96ba:/# getent hosts api
172.23.4.3 api
root@e9d0d67d96ba:/# getent hosts api.internal
root@e9d0d67d96ba:/#
Double checking:
root@e9d0d67d96ba:/# dig api
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> api
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37790
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
api. IN A
;; ANSWER SECTION:
api. 600 IN A 172.23.4.3
;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 13 09:36:00 UTC 2024
;; MSG SIZE rcvd: 86
root@e9d0d67d96ba:/# dig api.internal
; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> api.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7070
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 523005785d493d03e2bbf69f675c00038093858533afe10e (good)
;; QUESTION SECTION:
;api.internal. IN A
;; AUTHORITY SECTION:
. 600 IN SOA a.root-servers.net. nstld.verisign-grs.com. 2024121300 1800 900 604800 86400
;; Query time: 14 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 13 09:36:03 UTC 2024
;; MSG SIZE rcvd: 167
I know, but thank you for pointing it out. For this specific stack the API is still in testing and they just need it to have a direct port as well to bypass the frontend. This will be removed later. I just left it in the examples but it is indeed not needed here.
I haven’t used it for years, so I forgot that it doesn’t exactly work as you tried. I tested without compose first, that’s why I got a different result. Compose creates a custom network for the internal network on which the service name become an alias, and you can indeed use servicename.networkname, but the network name is not what you define in compose, but what you have on the host. There is no alias for the network name. So api.internal will not work, but api.${PROJECT_NAME}_${COMPOSE_NETWORK_NAME} would. I used variables as placeholders. You would need to replace those with the actual values. If your project name (folder name by default) is “myproject”, then the correct reference in the env variable would be:
This doesn’t make the network external, so compose will create and delete it, but you cannot use it in multiple compose projects so using the default prefix could be a better choice I guess.
If you use the default naming, you would need to know what the project name is before you start the project. When the folder contains special characters, that might not be obvious, but you can define the “name: projectname” parameter in the compose file if that helps. So these are the two options you have. Using the full network name in the domain, or using custom network aliases.
Of course, makes perfect sense, it is the network name, not “internal” referring to “stack local”. I feel stupid.
I did not chose to use internal as network name (would’ve preferred they kept default), but I stupidly assumed it was a compose keyword.
Yeah, but for us, it is exactly as you already suggested: We often use the same stack more once on a machine (dev + training env for example). So the project name varies depending on the environment and so does the resulting network. Renaming the network that way would throw both stacks on the same network.
But, using the network name (now that I know that indeed works ) is a much simpler way to do things.
We use a pre-compiler for everything docker/compose, so I’ll probably see if I can modify it to support the compose project name in that context.
) is a much simpler way to do things.