I have 2 different clusters running similar docker swarm services (GoCD server and several agents) on an Ubuntu machine deployed by a similar Ansible script. The issue exists only on Ubuntu 22.
On both clusters, I have a GoCD server single instance and multiple agents running on different nodes.
On the Ubuntu 20 cluster, the agents from all nodes are successfully connected to the GoCD server.
Docker version 20.10.24, build 297e128
On the Ubuntu 22 cluster, only the agents deployed on the same node as the one running the GoCD server can connect to the server. The other agents on the other nodes cannot connect to the GoCD server because they can’t access the port for some reason. They can however access the external host/port (host01/9100)
Docker version 24.0.7, build afdd53b
Additional information from the Ubuntu 22 cluster:
-
I’ve compared the ‘inspect’ result of the overlay docker network and ‘docker info’ between both clusters and besides the IDs it’s the same configuration
-
This is the result of running nmap from a service running on a different node (as you can see 8153 is not accessible):
root@bbcb24c97a41:/# nmap -p 8153 10.0.1.21
Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-03 16:38 GMT
Nmap scan report for 10.0.1.21
Host is up (0.000093s latency).
PORT STATE SERVICE
8153/tcp filtered quantastor
MAC Address: 02:42:0A:00:01:07 (Unknown)
Nmap done: 1 IP address (1 host up) scanned in 0.43 seconds
- And again, using the host IP and the external port, this time the port is accessible
root@bbcb24c97a41:/# nmap -p 9100 10.12.1.128
Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-03 16:38 GMT
Nmap scan report for XXX.XXX.XXX (10.12.1.128)
Host is up (0.00038s latency).
PORT STATE SERVICE
9100/tcp open jetdirect
Nmap done: 1 IP address (1 host up) scanned in 0.20 seconds
-
The firewall is disabled on all nodes:
➤ ufw status
Status: inactive -
Iptables result is the same across all nodes
➤ iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy DROP)
target prot opt source destination
DOCKER-USER all – anywhere anywhere
DOCKER-INGRESS all – anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
DROP all – anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (2 references)
target prot opt source destination
Chain DOCKER-INGRESS (1 references)
target prot opt source destination
ACCEPT tcp – anywhere anywhere tcp dpt:9100
ACCEPT tcp – anywhere anywhere state RELATED,ESTABLISHED tcp spt:9100
ACCEPT tcp – anywhere anywhere tcp dpt:9140
ACCEPT tcp – anywhere anywhere state RELATED,ESTABLISHED tcp spt:9140
RETURN all – anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
RETURN all – anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all – anywhere anywhere
DROP all – anywhere anywhere
RETURN all – anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all – anywhere anywhere