Greetings,
We need help with Docker networking, specifically with Docker Swarm.
We aim to set up a highly available (HA) Traefik Proxy as the entry point to our system from the public network over ports 80 and 443 (HTTP/HTTPS).
Traefik will be deployed on three manager nodes, and we will use a load balancer in front of our network to distribute traffic and if one manager node goes down, the others should handle incoming requests.
Our current VM setup is Docker Swarm with 3 manager nodes (Traefik) and 1 workers node (nginx, for now).
On manager nodes we deployed Traefik as Docker Swarm service with global deployment mode, and on worker node we have deployed nginx which will server our page.
Traefik on all nodes has ports 80 and 443 exposed to the host, so incoming HTTPS requests are passed to Traefik, which then forwards them to the nginx service on the worker node.
The problem we are facing is that only one Traefik instance successfully accepts connections and forwards them to the nginx service.
If we route all HTTPS requests to the Traefik instance on manager node A, everything works as expected. However, if we route requests to the Traefik instance on manager node B, the requests are not forwarded to the nginx service on the worker node, resulting in a “Bad Gateway” error.
I can ping the nginx service from all Traefik containers, but if I try to use telnet to connect to ports 80/443, the connection works only from the Traefik instance on manager node A.
Stack file:
version: '3.3'
services:
traefik:
image: traefik:v3.2.0
ports:
- mode: host
protocol: tcp
published: 443
target: 443
- mode: host
protocol: tcp
published: 80
target: 80
- mode: host
protocol: tcp
published: 5432
target: 5432
deploy:
mode: global
placement:
constraints:
- "node.role==manager"
labels:
- traefik.enable=true
- traefik.docker.network=traefik_traefik-public
- traefik.http.middlewares.https-redirect.redirectscheme.scheme=https
- traefik.http.middlewares.https-redirect.redirectscheme.permanent=true
- traefik.http.routers.traefik-public-http.rule=Host(`traefik.dev.si`)
- traefik.http.routers.traefik-public-http.entrypoints=http
- traefik.http.routers.traefik-public-http.middlewares=https-redirect
- traefik.http.routers.traefik-public-https.rule=Host(`traefik.dev.si`)
- traefik.http.routers.traefik-public-https.entrypoints=https
- traefik.http.routers.traefik-public-https.tls=true
- traefik.http.routers.traefik-public-https.service=api@internal
- traefik.http.services.traefik-public.loadbalancer.server.port=8080
volumes:
- /var/run/docker.sock:/var/run/docker.sock:ro
networks:
- traefik-public
networks:
traefik-public:
driver: overlay
ipam:
config:
- subnet: 10.0.10.0/24
Nginx stack file:
version: '3.3'
services:
nginx:
image: nginx
networks:
- traefik-public
container_name: nginx
volumes:
- app-tmp-public:/var/www/gms/current/public/tmp
logging:
driver: json-file
options:
max-size: 50m
max-file: 3
deploy:
placement:
constraints:
- "node.role!=manager"
labels:
traefik.enable: 'true'
traefik.http.routers.dev-web.rule: 'Host(`test.dev.net`)'
traefik.http.services.dev-web.loadbalancer.server.port: '80'
traefik.http.routers.dev-web.entrypoints: 'https'
traefik.http.routers.dev-web.tls: 'true'
update_config:
order: start-first
failure_action: rollback
delay: 5s
Docker Swarm Cluster setup
Manager A (Traefik :80 :443)
Manager B (Traefik :80 :443)
Manager C (Traefik :80 :443)
Worker A (nginx)
Overlay network: traefik-public
Works
Public → Manager A Traefik → Worker A nginx
Doesn’t works
Public → Manager B Treafik → Worker A nginx
Already tried but it doesn’t works:
- create overlay network outside of the stack
- map ports as ingress to host
- completly new VMs for managers
- ping MTU
Some screenshots of my testing.
On the left side of screenshot is manager A on which connectivity works and on right side is manager B where connectivity doesn’t works.
Ping, telnet, traceroute commands are exectued inside Traefik containers on manager node A/B.
