Multiple networks and priority?

I’ve had a doozy today.

All our compose stacks define 2 networks, 1 shared network - reverse_proxy (that is used to put the outside facing services on, for traefik reverse proxying) and then the internal network for the stack itself (interconnect).

Small example (everything unneeded removed):

services:
  frontend:
    labels:
      # Traefik lables
    environment:
      - API_PROXY_PASS=http://api:8080
    ports:
      - "80"
    networks:
      - internal
      - reverse_proxy

  api:
    ports:
      - "8080"
    networks:
      - internal

networks:
  internal:
  reverse_proxy:
    external: true

This morning I’ve had a ticket come in that there were problems with the API of one of our stacks.

After long searching, it turned out that the frontend got as IP for “api” the IP of a different stack that had it’s api service connected to the “reverse_proxy” proxy network, instead of the one from the “internal service”.

Is there any way to indicate what the priority is for dns resolving? The order in the compose file obviously does not seem to matter. Or perhaps the priority of the networks?

The simplest would obviously be to demand those connected to the reverse_proxy not to use a generic name but instead use [stackname]_[service} or something, but I’m not a big fan of that. hostname doesn’t work as it’s just an extra added to the dnsnames.

The problem you describe can only happen, if an api service is attached to the reverse-proxy network, so that name resolution sometimes returns the api service from the reverse-proxy network, and sometimes from the internal network.

If I remember right, the frontend can reach its backend in the same compose project using <service name>.<network name>, so in your case api.internal (I don’t recall if this only works for swarm services)

If the service name is not unique amongst all compose projects that use the same network, you need to introduce a network alias for the service in this specific network:

Thus, wort cast: add a network alias to the api service in the internal network, and make the frontend call the api service using the alias.

1 Like

Thank you for the reply.

I didn’t explicitly mention it, but there are indeed multiple APIs connected to that network and that is intended. The reverse_proxy network is where everything that has an outward facing port (and dns) connects to. Since we have multiple APIs facing outwards, more than one of those services is connected to this network.

I was hoping that the api.<network-suffix>. would work, because that would be awesome, but unfortunately api.internal on its own does not resolve so that must probably be swarm only as you suspected.

I did try a network alias, since would be visually more pleasing than renaming the service, except for the unfortunate fact that DNS-wise, this results in an alias, not a rename. It simply adds an extra hostname to that network, on top of the “api” one. Meaning that “api” will still be a known service within the reverse_proxy network.

I did not think about aliasing the API within the internal network though :slight_smile:
That would indeed be a much cleaner solution, but that would mean modifying nearly all our stacks instead of just the ones exposing an API to the public.

I guess I’ll ask the rest of the team what they think about it. At this point I don’t really see other alternatives either.

For future me/other people, this does work and looks fairly clean:

services:
  frontend:
    labels:
      # Traefik lables
    environment:
      - API_PROXY_PASS=http://api.internal:8080
    ports:
      - "80"
    networks:
      - internal
      - reverse_proxy

  api:
    ports:
      - "8080"
    networks:
      internal:
        aliases:
          - api.internal

networks:
  internal:
  reverse_proxy:
    external: true

Thank you again!

1 Like

I recommend checking it again, because that should indeed work without swarm. To make sure I’m right, I tested it.

Also do you have any reason to publish the API and frontend ports even without the reverse proxy? Because that is what

ports:
  - "8080"

does. It publishes the container port 8080 on an automatically chosen host port.

Well it doesn’t here:

root@e9d0d67d96ba:/# getent hosts api
172.23.4.3      api
root@e9d0d67d96ba:/# getent hosts api.internal
root@e9d0d67d96ba:/#

Double checking:

root@e9d0d67d96ba:/# dig api     

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> api
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37790
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
api.    IN      A

;; ANSWER SECTION:
api. 600 IN      A       172.23.4.3

;; Query time: 0 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 13 09:36:00 UTC 2024
;; MSG SIZE  rcvd: 86

root@e9d0d67d96ba:/# dig api.internal

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> api.internal
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 7070
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
; COOKIE: 523005785d493d03e2bbf69f675c00038093858533afe10e (good)
;; QUESTION SECTION:
;api.internal. IN        A

;; AUTHORITY SECTION:
.                       600     IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2024121300 1800 900 604800 86400

;; Query time: 14 msec
;; SERVER: 127.0.0.11#53(127.0.0.11) (UDP)
;; WHEN: Fri Dec 13 09:36:03 UTC 2024
;; MSG SIZE  rcvd: 167

I know, but thank you for pointing it out. For this specific stack the API is still in testing and they just need it to have a direct port as well to bypass the frontend. This will be removed later. I just left it in the examples but it is indeed not needed here.

I haven’t used it for years, so I forgot that it doesn’t exactly work as you tried. I tested without compose first, that’s why I got a different result. Compose creates a custom network for the internal network on which the service name become an alias, and you can indeed use servicename.networkname, but the network name is not what you define in compose, but what you have on the host. There is no alias for the network name. So api.internal will not work, but api.${PROJECT_NAME}_${COMPOSE_NETWORK_NAME} would. I used variables as placeholders. You would need to replace those with the actual values. If your project name (folder name by default) is “myproject”, then the correct reference in the env variable would be:

    environment:
      - API_PROXY_PASS=http://api.myproject_internal:8080

You can also rename the network externaly

networks:
  internal:
    name: uniquename

And use this:

    environment:
      - API_PROXY_PASS=http://api.uniquename:8080

This doesn’t make the network external, so compose will create and delete it, but you cannot use it in multiple compose projects so using the default prefix could be a better choice I guess.

If you use the default naming, you would need to know what the project name is before you start the project. When the folder contains special characters, that might not be obvious, but you can define the “name: projectname” parameter in the compose file if that helps. So these are the two options you have. Using the full network name in the domain, or using custom network aliases.

Of course, makes perfect sense, it is the network name, not “internal” referring to “stack local”. I feel stupid.
I did not chose to use internal as network name (would’ve preferred they kept default), but I stupidly assumed it was a compose keyword.

Yeah, but for us, it is exactly as you already suggested: We often use the same stack more once on a machine (dev + training env for example). So the project name varies depending on the environment and so does the resulting network. Renaming the network that way would throw both stacks on the same network.

But, using the network name (now that I know that indeed works :smirk:) is a much simpler way to do things.
We use a pre-compiler for everything docker/compose, so I’ll probably see if I can modify it to support the compose project name in that context.

Thank you both so much for the assistance!