Docker Community Forums

Share and learn in the Docker community.

Mutiple interface, policy based routes in docker swarm overlay network not reachable

swarm

(Ts4096) #1

Issue type: Published port is not reachable in swarm.
OS Version/build: RHEL 7.4,
Server Version: 1.13.1
Storage Driver: overlay2
Backing Filesystem: xfs

App version
Steps to reproduce
On my server there are 4 nics,
p3p1, p3p2, em1, em2
while creating swarm p3p1 ip is used as --advertise-addr.
em1 and p3p1 are in different subnet
for em1 and em2 - there IP’s are in same subnet. they have policy based routes configured

[root@node1 ~]# ip rule show
150: from 10.209.194.247 lookup 1
[root@node1 ~]# ip route show tab 1
default via 10.209.192.1 dev em1
10.209.192.0/22 dev em1 scope link

docker service create --name srv1 --replicas 2 --publish published=8080,target=80 nginx:alpine
curl 127.0.0.1:8080 <---- this works
but 10.209.194.247 this ip don’t work.

when I remove rules for IP these request from 10.209.194.247 works, If I create docker instance outside the swarm it also works.
The combination of swarm + policy route + overlay network is not working. I think it only uses p3p1 (and loopback)
What additional steps we need to ensure to make ip rule work?


(Ts4096) #2

My initial finding is because of masquerade in iptables only ips of first interface are able to reply from published port.
Has anyone configured same subnet on multiple nic on host and published docker service port via all nics?