Portainer version: 2.18.3
I deployed a swarm with a few servers: (image 1st and 2nd part)
All servers are located in the same network (running Ubuntu 20.04 or 22.04) and all traffic is allowed between these hosts by ufw rules (ufw allow from x). I also mange the router behind this network and there is no specific rules for these hosts. nmap to port 9001 is open in all servers from the manager server.
All services are deployed using stacks. I can only have full access (console, stats) to services deployed in K and L (see image 3rd part)
Also only containers located at K or L are listed (see image 4th part)
Another problem is that services within the same stack, but placed across different hosts (for example in T, W and B) fail to communicate between them. In other words, a container in T can not ping a container in W even though the are in the same docker network (specified in the stack). Only works if all services inside a stack are placed in the same host.
I suspect there might be something related to the network as I have seen similar reports when the hosts are not in the same network. But here the servers are physically connected to the same router.
This is a simple map of this network:
K, L -----> [ Switch 1 ] --------\
|----[ Router ]
T, B, W ----> [ Switch 2 ] ------/
I have tried disabling UFW in all those servers and wait a few minutes but nothing changes.
What could be causing this situation? How can I fix it?
NOTE: I have deployed similar scenarios before several times without issues (usually using several VPS in AWS and other cloud services)