Hello,
I have two hosts (two VMs), where each host will run two containers each, one app and one database (YugabyteDB). The containers will be deployed with two different docker compose files. The two hosts have IPs 193.168.120.243 and 193.168.120.244.
The two different database containers will be connected using an overlay network created by docker swarm init. I have no plan to create swarm services. My plan is only to enable communication between the database containers on the overlay network.
Hence, I have the following networks:
- external_network: this is the overlay network between the two database containers deployed on the two hosts. This ends up on a 10.0.1.0/24 subnet.
- internal_network: this is a standard bridge network between the app and database containers. This ends up on a 172.19.0.0/16 subnet.
I have managed to create the swarm cluster and I can successfully communicate between the database containers on the overlay network. When inspecting the database containers, I can see that they are part of both the external 10.0.1.0/24 subnet and the internal 172.19.0.0/16 subnet. The IP assigned to the database container on the external network is 10.0.1.2 and on the internal network 172.19.0.2.
So far so good.
Now to the problem:
It seems that the overlay network refuses any external connection to it. There are two connections to the database containers I need:
- I want to be able to access the database container (running a YubabyteDB UI) on port 15433 and also two special dashboards available on ports 7000 and 9000 from each of the hosts. For example: 193.168.120.243 → 10.0.1.2 on port 15433.
- Each app container needs to communicate with the database containers on ports 5433. For example: 172.19.0.3 → 172.19.0.2 on port 5433.
Both of these connections are refused and I cannot understand why. All ports needed are forwarded as required in the docker compose file. With every attempt, the access into the overlay network from either the host or another container seems impossible. Only access from nodes within the overlay network has been successful.
For example, this is how it looks when I try to reach the database UI from the host on port 15433:
curl -v 192.168.120.243:15433
* Trying 192.168.120.243:15433...
* connect to 192.168.120.243 port 15433 failed: Connection refused
* Failed to connect to 192.168.120.243 port 15433: Connection refused
* Closing connection 0
curl: (7) Failed to connect to 192.168.120.243 port 15433: Connection refused
This is the message get when trying to reach the database on port 5433 from the app container:
ERROR | app.database.session:<module>:36 - Failed attempt to connect to the database: (psycopg2.OperationalError) connection to server at "db" (172.19.0.2), port 5433 failed: Connection refused
However, if I log into the running database container and run curl -v 10.0.1.2:15433, it can access the UI with no issues. Hence, the issue is only present from outside the overlay network. The port forwarding in the docker compose file does not seem to work.
If we only look from the perspective of host with IP 193.168.120.243. Here is the compose file:
name: my_project
networks:
internal_network:
name: internal_network
driver: bridge
external_network:
name: external_network
driver: overlay
attachable: true
services:
app:
image: app-image:1.0
container_name: my_app
hostname: host_name
networks:
- internal_network
restart: always
environment:
DATABASE_URL: postgresql://my_user:123my_password@db:5433/my_db
ports:
- 80:8000
depends_on:
- db
db:
image: yugabytedb/yugabyte:2025.1.0.1-b3
container_name: my_container_name_1
hostname: my_hostname_1
networks:
- internal_network
- external_network
restart: always
command: [ "bin/yugabyted",
"start",
"--background=false",
"--advertise_address=my_container_name_1",
"--cloud_location=my_cloud.my_region_1.my_zone_1" ]
environment:
POSTGRES_DB: my_db
POSTGRES_USER: my_user
POSTGRES_PASSWORD: 123my_password
ports:
- 7000:7000
- 7100:7100
- 9000:9000
- 9100:9100
- 15433:15433
- 5433:5433
- 9042:9042
My questions:
- Why can I not access the overlay network from outside the overlay network?
- Should not the ports section in the compose file open them up from both the host as well as other containers?
- How can I fix this?
I am running v2.39.4 of docker compose.