I have containers running in swarm mode on separate devices/nodes, including a Postgres instance.
Postgres is configured with listen_addresses='*'
. Inspecting the logs on the instance (docker service logs xrs_db
) demonstrates that it’s listening on 0.0.0.0
, port 5432:
root@traefik:~# docker service logs xrs_db
xrs_db.1.gl8lhg147726@database |
xrs_db.1.gl8lhg147726@database | PostgreSQL Database directory appears to contain a database; Skipping initialization
xrs_db.1.gl8lhg147726@database |
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.310 UTC [1] LOG: starting PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.310 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.310 UTC [1] LOG: listening on IPv6 address "::", port 5432
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.311 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.313 UTC [29] LOG: database system was shut down at 2022-06-20 22:55:01 UTC
xrs_db.1.gl8lhg147726@database | 2022-06-20 22:55:46.317 UTC [1] LOG: database system is ready to accept connections
On the database node, docker ps
confirms that the postgres container has port 5432 open:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
c0e0691cc03b postgres:14.0 "docker-entrypoint.s…" About a minute ago Up About a minute (healthy) 5432/tcp xrs_db.1.gl8lhg147726le4n9e9ov1l6h
1fe5cefcd4d3 bash:4.4 "docker-entrypoint.s…" About a minute ago Up About a minute xrs_bash-same-node.1.vgwzef25hhn9ckzvfjhl3bbk0
I’ve made sure that the firewall is configured to allow incoming connections on port 5432:
root@database:~# ufw status
Status: active
To Action From
-- ------ ----
22 ALLOW Anywhere
5432/tcp ALLOW Anywhere
22 (v6) ALLOW Anywhere (v6)
5432/tcp (v6) ALLOW Anywhere (v6)
I’ve also disabled the firewall on my cloud hosting provider.
If I log into the postgres container itself and run psql it connects without issue (as you would expect):
root@database:~# docker exec -it c0e0691cc03b psql --host=db --port=5432 --username=postgres --dbname=xr
Password for user postgres:
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.
xr=#
If I log into another container on the same node I can see that the port is open:
root@database:~# docker exec 1fe5cefcd4d3 nc -zv db 5432
db (10.0.35.5:5432) open
If I switch to a different node (i.e. a different machine), I can ping that device:
root@worker:~# docker exec e1570f13a6bf ping db -c1
PING db (10.0.35.5): 56 data bytes
64 bytes from 10.0.35.5: seq=0 ttl=64 time=0.305 ms
…but I get timeout when trying to connect to port 5432:
root@worker:~# docker exec e1570f13a6bf nc -zv db 5432
nc: db (10.0.35.5:5432): Operation timed out
Any idea what I could be missing?
My docker compose file:
version: '3.8'
x-default-opts:
&default-opts
logging:
options:
max-size: '1m'
max-file: '3'
services:
db:
<<: *default-opts
image: postgres:14.0
secrets:
- POSTGRES_USER
- POSTGRES_PASS
- POSTGRES_DB
environment:
POSTGRES_USER_FILE: /run/secrets/POSTGRES_USER
POSTGRES_PASSWORD_FILE: /run/secrets/POSTGRES_PASS
POSTGRES_DB_FILE: /run/secrets/POSTGRES_DB
PGDATA: /var/lib/postgresql/data
volumes:
- /database-files:/var/lib/postgresql/data
networks:
- backend
ports:
- 5432:5432
healthcheck:
test: "exit 0"
deploy:
replicas: 1
placement:
constraints:
- node.role == worker
- node.labels.role==database
restart_policy:
condition: on-failure
bash-same-node:
<<: *default-opts
image: bash:4.4
stdin_open: true
tty: true
networks:
- backend
deploy:
replicas: 1
placement:
constraints:
- node.role == worker
- node.labels.role==database
restart_policy:
condition: on-failure
bash-different-node:
<<: *default-opts
image: bash:4.4
stdin_open: true
tty: true
networks:
- backend
deploy:
replicas: 1
placement:
constraints:
- node.role == worker
- node.labels.role==worker
restart_policy:
condition: on-failure
networks:
backend:
driver: overlay
attachable: true
secrets:
POSTGRES_USER:
external: true
POSTGRES_PASS:
external: true
POSTGRES_DB:
external: true