Overlay network in computer with two IPs

I’m trying to use an attached overlay network.

The version of the nodes is:

ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
pd7f0gm1c4kx28p91zos4or6l     yamato    Down      Active                          23.0.1
u8hn5nmb001fz6yzsf8clug39     iscandar   Ready     Active                          23.0.1
l5gu7053pd5gm30grswdwibac     suisei     Ready     Active                          23.0.2
2pg6o8tsd0g7ps5bw9erxazp6 *   tsuki      Ready     Active         Leader           23.0.2

The network was created with the following command:

docker network create -d overlay --attachable overlay_proxy

I also ran a sample service:

docker service create --name whoami --network overlay_proxy containous/whoami

As I understand it, any container attached to this network should be able to curl whoami:80, but this doesn’t happen, all devices in the swarm (except the one running the whoami task) give me the following error:

$ docker run --rm -it --network overlay_proxy nginx getent hosts whoami
10.0.1.16        whoami
$ docker run --rm -it --network overlay_proxy nginx curl whoami
curl: (28) Failed to connect to whoami port 80: Connection timed out

As you can see, it resolves the VIP correctly. The problem must be routing the packets or something.

Expected output:

docker run --rm -it --network overlay_proxy nginx curl whoami
Hostname: b9e7a9faa312
IP: 127.0.0.1
IP: 10.0.1.17
IP: 172.20.0.3
RemoteAddr: 10.0.1.18:52708
GET / HTTP/1.1
Host: whoami
User-Agent: curl/7.74.0
Accept: */*

What am I doing wrong? How can I debug this?

Journalctl of one of the nodes:

abr 04 14:05:43 iscandar dockerd[780]: time="2023-04-04T14:05:43.640302792+02:00" level=info msg="ignoring event" container=19a71185d9e0997a0b23da1643c199b62c51d1e5228f80d139734025bf282fd4 module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
abr 04 14:05:44 iscandar dockerd[780]: time="2023-04-04T14:05:44.284593161+02:00" level=warning msg="Error (Unable to complete atomic operation, key modified) deleting object [endpoint rpszdau3mwhv4pi3qgzf88gbh e5aa5bab55b31b535c9e0a6ebef2271256d6081a689572bd7e95ad84ee83ab8b], retrying...."
abr 04 14:05:44 iscandar dockerd[780]: time="2023-04-04T14:05:44.336776228+02:00" level=info msg="initialized VXLAN UDP port to 4789 "
abr 04 14:05:44 iscandar dockerd[780]: time="2023-04-04T14:05:44.337044744+02:00" level=error msg="failed removing service binding for 0eb5521969bc8c712ea1be50dbf0701989440a2f8e3d0eaf24898dc9a6e50e71 epRec:{whoami.1.n5jd972ks2ql5hphmnpcjh20u whoami kekrt31a34blj55od7ug93fcv

I’m using a recently installed Debian, without touching the firewall, so ports should be open

It works if I publish the ports using -p80:80, but then I can only have one service using that port…

version: "3.8"
networks:
  overlay_proxy:
    external: True
    driver: overlay
services:
  whoami:
    image: containous/whoami
    ports:
      - 80:80
    networks:
      overlay_proxy:

Apparently, two of the nodes can communicate between them. n1 can communicate n2 and vice versa, but n3 (the swarm Leader) is uncommunicated. How is that possible?

n1 and n2 are connected “directly” to the router, but n3 is just behind a switch

I checked with ncat that all ports were reachable. Now, with wireshark, I’m seing some errors with filter udp.port == 4789, in the connection n1<->n3, but in the packaged TCP.

It starts the tcp handshake, with succesfull syn and syn-ack, but the final ack seems to be missing. This is strange because I’m running wireshark in the client…

But It seems to work fine connecting to the other node

I checked INSIDE the container with tcpdump and the problem seems to be that, even though the server (n3) sends the SYN+ACK of the handshake, the client’s container (n1) doesn’t receive it and so curl doesn’t respond with the HTTP packet. Why does the SYN+ACK go missing between the VXLAN and the container?

The ip of the containers is 10.0.3.28 and 10.0.3.5, while the IPs specified in the packet are 10.0.3.9 and 10.0.3.3, which correspond to their respective lb-overlay_proxy container (according to docker inspect).

Nevertheless, if it works with the other node but not with the Leader, why the failure seems to be on n1?? I mean, n1 seems to receive the udp packet with the SYN+ACK, but it does not “pass it” to the container using curl in n1. Why? Is this a bug?

Upon more inspection, the response from the Leader seems to have a different IP (192.168.1.11). This is because the Leader has two IP addresses.

Perhaps the client drops this package because it sent it to 192.168.1.32, but it receives a response from 192.168.1.31? It shouldn’t matter because the VXLAN is the same

Yes! That seemed to be the problem. Deleting the secondary IP address seemed to work. Now, how can I keep both IPs but making sure that it replies with the IP it received the package in?

Why does it send a response using a different IP?

Had the affected node always had a secondary IP?

From what I remember it is not possible to initialize a swarm on a host with multiple interfaces/ips, unless --advertise-addr is specified. It will be used for cluster management and overlay data. Though, If --data-path-addr is specified the overlay data will use that inteface/ip.

The same applies to nodes that join a swarm. If no advertise address is provided, the source ip used to reach the existing manger node for the join operation is used as advertise-addr.

You might want to inspect the nodes and make sure all the nodes are registered in the expected ip-range.

I’m still having the same problem. Is there any way to modify the --data-path-addr without creating a new swarm?

Btw: Modifying daemon.json did not work. I thought I solved it, but it was just because I also deleted and re-added the other IP.

Easy for the worker nodes: remove each node and re-join it with added parameter. With 3 manager nodes, or more this approach could be used for manager nodes as well.

Though, For a single manager node: there is no way other than resolving the swarm and re-initializing a new one.

I had a single manager node. I promoted the two other nodes to manager (and the demoted them), left the swarm and joined again with correct parameters.

This solved the problem, I hope permanently