Docker Compose network: cannot use default docker gateway to connect to host machine from container, unless a custom network is manually specified

Docker Engine Compose
,
1

I have a simple compose file at ~/tmp/:

version: '3.9'

networks:
  network1:
    name: pg-net
    attachable: false
    ipam:
      driver: default
      config:
        - subnet: 172.20.0.0/16
          ip_range: 172.20.2.0/24
          gateway: 172.20.0.1

services:
  postgres:
    image: postgres:14.1-bullseye
    hostname: postgres
    environment:
      POSTGRES_PASSWORD: ****
    ports:
      - "5433:5432"
    #networks: # for the moment, keep the default network that will be build during the start up of the container
    #  - network1

when connecting to my container and trying to connect to a dummy database on the host machine, the psql command times out:

$ dc down -v && dc up -d && dc exec postgres bash

root@postgres:/# psql -d postgres://postgres@172.17.0.1:5432/dummydb
psql: error: connection to server at "172.17.0.1", port 5432 failed: Connection timed out
	Is the server running on that host and accepting TCP/IP connections?

but if I uncomment the two last lines of the compose file, it works:

root@postgres:/# psql -d postgres://postgres@172.20.0.1:5432/dummydb
Password for user postgres:  *********************
psql (14.0 (Debian 14.0-1.pgdg110+1), server 12.4 (Ubuntu 12.4-1.pgdg18.04+1))
Type "help" for help.

dummydb=#

but the strangest is that if I use the standard docker0 IP address, which is the default docker Gateway 172.17.0.1 it also works (Iā€™m really surprised there !!):

root@postgres:/# psql -d postgres://postgres@172.17.0.1:5432/dummydb
Password for user postgres:  *********************
psql (14.0 (Debian 14.0-1.pgdg110+1), server 12.4 (Ubuntu 12.4-1.pgdg18.04+1))
Type "help" for help.

dummydb=#

Here is my custom network:

$ docker network inspect -f '{{json .}}' pg-net | jq .
{
  "Name": "pg-net",
  "Id": "4ad4a93eef77fff6a2e3972b3dde4b2a990cf96dda0df00a11fac62de86c1507",
  "Created": "2022-01-15T21:39:00.681972675+01:00",
  "Scope": "local",
  "Driver": "bridge",
  "EnableIPv6": false,
  "IPAM": {
    "Driver": "default",
    "Options": null,
    "Config": [
      {
        "Subnet": "172.20.0.0/16",
        "IPRange": "172.20.2.0/24",
        "Gateway": "172.20.0.1"
      }
    ]
  },
  "Internal": false,
  "Attachable": true,
  "Ingress": false,
  "ConfigFrom": {
    "Network": ""
  },
  "ConfigOnly": false,
  "Containers": {
    "302408154a2b84b75eda328e76c6de3676aq4c042038aa83fb7e3aa6fc640e10": {
      "Name": "tmp_postgres_1",
      "EndpointID": "f05c7d0faf9e415d8adde05de953d41beea5dd12605ebcf42d49f973bb03f71a",
      "MacAddress": "02:42:ac:ad:44:00",
      "IPv4Address": "172.20.2.0/16",
      "IPv6Address": ""
    }
  },
  "Options": {},
  "Labels": {
    "com.docker.compose.network": "pg-net",
    "com.docker.compose.project": "tmp",
    "com.docker.compose.version": "1.27.4"
  }
}

(( please also note that even Iā€™ve set attachable: false is has been switched to true, I donā€™t know whyā€¦ ))

and hereā€™s the network when the two last lines of the compose file were actually commented, so when the network was auto-magically build when spinning up the container:

$  docker network inspect -f '{{json .}}' tmp_default | jq .
{
  "Name": "tmp_default",
  "Id": "04b249a2367007baba5187130587823d7e027f49d7424a8423e7a6f69e754601",
  "Created": "2022-01-15T21:45:28.507326828+01:00",
  "Scope": "local",
  "Driver": "bridge",
  "EnableIPv6": false,
  "IPAM": {
    "Driver": "default",
    "Options": null,
    "Config": [
      {
        "Subnet": "192.168.62.0/20",
        "Gateway": "192.168.62.1"
      }
    ]
  },
  "Internal": false,
  "Attachable": true,
  "Ingress": false,
  "ConfigFrom": {
    "Network": ""
  },
  "ConfigOnly": false,
  "Containers": {
    "1157fca611cb189e85f0d878a6a6bdd042d5d625e7486584feda2df77de10229": {
      "Name": "tmp_postgres_1",
      "EndpointID": "6f87632562208aca2d3193156faee985a3d06a1ca829af6e56f814a8a66b0382",
      "MacAddress": "02:42:c0:a9:23:02",
      "IPv4Address": "192.168.62.2/20",
      "IPv6Address": ""
    }
  },
  "Options": {},
  "Labels": {
    "com.docker.compose.network": "default",
    "com.docker.compose.project": "tmp",
    "com.docker.compose.version": "1.27.4"
  }
}

Iā€™m wondering why does the connection using 172.17.0.1 suddenly (!!) work when configuring manually a totally different network?!

Info:

$ docker-compose --version
docker-compose version 1.27.4, build 40524192

$ docker --version
Docker version 20.10.12, build e91ed57

$ uname -mor
5.4.0-91-generic x86_64 GNU/Linux

OS: Ubuntu 18.04.6 LTS

2

Docker0 is just an interface on you host and is reachable like any ethx or other interface on the host would be. I am not surprised traffic that leaves the container network is able to access the ip .

Though, I am actualy surprised, why it didnā€™t work in the first example. I would have expected it to work in both situations.

3

May it be that this interface (docker0 : 172.17.0.1) is not reachable if the container is not in the same subnet?

4

I noticed the same before I read this part of your message. I managed to make it false somehow but I didnā€™t know how. It turned out it works with Docker Compose v2 but doesnā€™t work with Docker Compose v1.29.2 and one time I wrote docker compose up -d (v2) instead of docker-compose up -d (v1).

I couldnā€™t reproduce your connection issue and I also have Ubuntu 18.04.6 and Docker 20.10.12. I tried with Docker Compose v2.2.2, Docker Compose v1.29.2 and Docker Compose v1.27.4.

None of your examples added your container to that subnet. It could be your firewall on the host or something with how you run the postgres server on the host. I ran the host version in a container too using the host network

docker run -d -e POSTGRES_PASSWORD=password --net host postgres:14.1-bullseye
5

Well, it was something related to the firewall, indeed, but I donā€™t really fully understand itā€¦

Iā€™ll try to explain it.

In the first situation, with no manually defined network, I have set up a firewall rule (ufw) to match the docker default range on the host:

5432                       ALLOW IN    172.17.0.0/16

So, as the container is having the randomly assigned 172.20.0.1 Gateway, this seems logic to me that it cannot reach the local postgres service using the default one 172.17.0.1 (confirmation at the very bottom of this post).
(( also, listen_addresses = '*' in postgresql.conf and I allowed an even larger set of connections from this range 172.0.0.0/8 in pg_hba.conf on the local postgresql. These are the only modifications. ))

In the second example, when actually using the custom network, I have to change the firewall rule so that it matches the network, as follow (otherwise it doesnā€™t work):

5432                       ALLOW IN    172.20.0.0/16

Now it works, either with 172.17.0.1 or 172.20.0.1ā€¦ butā€¦
ā€¦after some more investigations, I finally noticed in this second situation, that I can successfully connect to the host postgres with every single Gateway address of every running services; this includes 5 other IPs which are actually not listed in ufw (but these are covered by the large range in pg_hba.conf).

Fun fact: this is totally random, but I do have an other service running on this Gateway 192.168.208.1 and I can connect to the host postgres using this IP even so there is strictly no firewall rule for this IP, neither an entry in pg_hba.conf. So this one should be blocked, but itā€™s notā€¦

Finally, in the first situation, if I change the ufw rule to match the randomly assigned Gateway (letā€™s say .22.) as follows: ALLOW IN 172.22.0.0/16 I can also reach the local postgres using whatever Gateway of any of the running services, including the default Gateway, exactly as in the previously described situation.

So to make it work, I have to allow an ufw rule matching the actual container Gateway, not the default 172.17.0.0/16, in order to access postgres. And in that case, pg is accessible through every single existing docker Gateway (even so they are not explicitly allowed by ufw (!!) Therefore, I donā€™t fully understand what exactly happens, this is some kind of black magic to me for the momentā€¦) !

Lastly, if I set up a more restrictive 172.17.0.0/16 in pg_hba.conf and trying to connect, it confirm what I originally thought:

# notice the .17. here, but the current container Gateway is .22.
root@postgres:/# psql -d postgres://postgres@172.17.0.1:5432/postgres
psql: error: connection to server at "172.22.0.1", port 5432 failed: FATAL:  no pg_hba.conf entry for host "172.22.0.2", user "postgres", database "postgres", SSL on
connection to server at "172.22.0.1", port 5432 failed: FATAL:  no pg_hba.conf entry for host "172.22.0.2", user "postgres", database "postgres", SSL off

so pg refuses the connection from the actual container Gateway, not the docker default, which I gave.

6

I always wanted to play with ufw. You gave me the reason so tried it, although I suspected the result even before that. I had to find out what gave you the output you quoted in your post. I found that ufw status verbose shows ALLOW IN and simple ufw status shows only ALLOW.

This rule

5432                       ALLOW IN    172.17.0.0/16

can be created this way:

ufw allow from 172.17.0.0/16 to any port 5432

It means you allow access FROM 172.17.0.0/16 to port 5432 on ANY IP address. When you changed the rules

5432                       ALLOW IN    172.20.0.0/16

which can be created this way:

ufw allow from 172.20.0.0/16 to any port 5432

it means you allow access FROM 172.20.0.0/16 to port 5432 on any IP address. This is why you could allow each gateway because you specified only the source and allowed each target. If you want to allow connection from any network to a specific IP address, you can do this:

ufw allow from any to 172.17.0.1 port 5432

Then the status is:

172.17.0.1 5432              ALLOW IN    Anywhere

and finally allowing access from a specific network to a specific IP address

ufw allow from 172.20.0.0/16 to 172.17.0.1 port 5432

172.17.0.1 5432              ALLOW IN    172.20.0.0/16
1 Like
7

FROM 172.23ā€¦? isnā€™t that a typo? donā€™t you mean 172.17ā€¦ here?

Btw, I usually use ufs status numbered (which is as verbose as verbose + it adds a number, number which can be used to delete a rule for example, this is practical).

And thank your for the clear explanations, I think you place the last piece of the puzzle I was missing. Iā€™ll play around with that to fully grab and understand whatā€™s happening.

The only thing left that I donā€™t understand, is the fact that even without an entry in pg_hba.conf. postgres is successfully responsive on the 192.168.208.1 interfaceā€¦

1 Like
8

It was a typo but bigger than you thought :slight_smile: Thanks. I fixed it in my original post. I wanted to write 172.20.0.0/16 but I tested in a different network where I had 172.23.0.0/16. Only the ufw status was right, sorry. It is really 172.20.0.0/16 but it could be anything

9

Ah okay, but then the line before:

should also be 172.20ā€¦ right? Not 172.17ā€¦?

and then the following is a duplicate because you use the same IP:

which is now consistent. I guess. Maybe you wanted to place .17. everywhere on the first example?

1 Like
10

Ohā€¦ let me recheck my whole post. It was late when I wrote it.

@swissknight I think you were right when you thought I wanted to write 17 and not 23ā€¦ It seems I am still tired. but it is so good that you notice these errors and not just copy the wrong commands. Can you still see any potential mistake?

11

No, the rest sounds great!
Iā€™m really grateful for all the time you spent!
May this thread help others in the future. :wink: