Swarm networking - can connect to port from another container on same node but not from different node

I have containers running in swarm mode on separate devices/nodes, including a Postgres instance.

Postgres is configured with listen_addresses='*'. Inspecting the logs on the instance (docker service logs xrs_db) demonstrates that it’s listening on 0.0.0.0, port 5432:

root@traefik:~# docker service logs xrs_db
xrs_db.1.gl8lhg147726@database    | 
xrs_db.1.gl8lhg147726@database    | PostgreSQL Database directory appears to contain a database; Skipping initialization
xrs_db.1.gl8lhg147726@database    | 
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.310 UTC [1] LOG:  starting PostgreSQL 14.0 (Debian 14.0-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.310 UTC [1] LOG:  listening on IPv4 address "0.0.0.0", port 5432
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.310 UTC [1] LOG:  listening on IPv6 address "::", port 5432
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.311 UTC [1] LOG:  listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.313 UTC [29] LOG:  database system was shut down at 2022-06-20 22:55:01 UTC
xrs_db.1.gl8lhg147726@database    | 2022-06-20 22:55:46.317 UTC [1] LOG:  database system is ready to accept connections

On the database node, docker ps confirms that the postgres container has port 5432 open:

CONTAINER ID   IMAGE           COMMAND                  CREATED              STATUS                        PORTS      NAMES
c0e0691cc03b   postgres:14.0   "docker-entrypoint.s…"   About a minute ago   Up About a minute (healthy)   5432/tcp   xrs_db.1.gl8lhg147726le4n9e9ov1l6h
1fe5cefcd4d3   bash:4.4        "docker-entrypoint.s…"   About a minute ago   Up About a minute                        xrs_bash-same-node.1.vgwzef25hhn9ckzvfjhl3bbk0

I’ve made sure that the firewall is configured to allow incoming connections on port 5432:

root@database:~# ufw status
Status: active

To                         Action      From
--                         ------      ----
22                         ALLOW       Anywhere                  
5432/tcp                   ALLOW       Anywhere                  
22 (v6)                    ALLOW       Anywhere (v6)             
5432/tcp (v6)              ALLOW       Anywhere (v6)             

I’ve also disabled the firewall on my cloud hosting provider.

If I log into the postgres container itself and run psql it connects without issue (as you would expect):

root@database:~# docker exec -it c0e0691cc03b psql --host=db --port=5432 --username=postgres --dbname=xr
Password for user postgres: 
psql (14.0 (Debian 14.0-1.pgdg110+1))
Type "help" for help.

xr=# 

If I log into another container on the same node I can see that the port is open:

root@database:~# docker exec 1fe5cefcd4d3 nc -zv db 5432
db (10.0.35.5:5432) open

If I switch to a different node (i.e. a different machine), I can ping that device:

root@worker:~# docker exec e1570f13a6bf ping db -c1
PING db (10.0.35.5): 56 data bytes
64 bytes from 10.0.35.5: seq=0 ttl=64 time=0.305 ms

…but I get timeout when trying to connect to port 5432:

root@worker:~# docker exec e1570f13a6bf nc -zv db 5432
nc: db (10.0.35.5:5432): Operation timed out

Any idea what I could be missing?

My docker compose file:

version: '3.8'

x-default-opts:
  &default-opts
  logging:
    options:
      max-size: '1m'
      max-file: '3'

services:

  db:
    <<: *default-opts
    image: postgres:14.0
    secrets:
      - POSTGRES_USER
      - POSTGRES_PASS
      - POSTGRES_DB
    environment:
      POSTGRES_USER_FILE: /run/secrets/POSTGRES_USER
      POSTGRES_PASSWORD_FILE: /run/secrets/POSTGRES_PASS
      POSTGRES_DB_FILE: /run/secrets/POSTGRES_DB
      PGDATA: /var/lib/postgresql/data
    volumes:
      - /database-files:/var/lib/postgresql/data
    networks:
      - backend
    ports:
      - 5432:5432
    healthcheck:
      test: "exit 0"
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == worker
          - node.labels.role==database
      restart_policy:
        condition: on-failure

  bash-same-node:
    <<: *default-opts
    image: bash:4.4
    stdin_open: true
    tty: true
    networks:
      - backend
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == worker
          - node.labels.role==database
      restart_policy:
        condition: on-failure

  bash-different-node:
    <<: *default-opts
    image: bash:4.4
    stdin_open: true
    tty: true
    networks:
      - backend
    deploy:
      replicas: 1
      placement:
        constraints:
          - node.role == worker
          - node.labels.role==worker
      restart_policy:
        condition: on-failure

networks:
  backend:
    driver: overlay
    attachable: true

secrets:
  POSTGRES_USER:
    external: true
  POSTGRES_PASS:
    external: true
  POSTGRES_DB:
    external: true

I worked it out. I needed to open Docker Swarm ports on all nodes, not just the manager node. So in practice that meant running the following:

ufw allow 2377/tcp
ufw allow 7946/tcp
ufw allow 7946/udp
ufw allow 4789/udp

In addition, there was no need to open port 5432 on the machine that hosted the postgres container, because network traffic is routed by Docker via the above ports.