Swarm over Wireguard VPN multi cloud

I try to do the same setup with wireguard and two cloud provider. All route are ok, I create my cluster and swarm can communicate through UDP and TCP through internal LAN.

Sam problem when services try to communicate on the same overlay networks, the dns docker resolve the ip but the ping doesnt reach the target.

From container with rabbitmq service

ping tasks.rabbitmq
PING tasks.rabbitmq (10.40.0.105) 56(84) bytes of data.
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=3 ttl=64 time=0.076 ms
^C
--- tasks.rabbitmq ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2022ms
rtt min/avg/max/mdev = 0.061/0.118/0.219/0.071 ms

From the container on the other network through wireguard

ping tasks.rabbitmq
PING tasks.rabbitmq (10.40.0.105): 56 data bytes
^C
--- tasks.rabbitmq ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss

I have debug a little further with tcpdump and udp port 4789. Because swarm use vxlan techno to send ip packet to another host, I saw the echo request and echo reply in UDP packet:
CF below
10.110.1.130 is the worker where the sender icmp container is
10.10.3.182 is the worker where the target container is

sudo tcpdump -i any -n -p udp port 4789

13:55:38.193961 IP 10.110.1.130.45777 > 10.10.3.182.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.35 > 10.0.40.100: ICMP echo request, id 99, seq 7, length 64
13:55:38.194083 IP 10.10.3.182.53901 > 10.110.1.130.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.100 > 10.0.40.35: ICMP echo reply, id 99, seq 7, length 64

And after on the worker where the sender icmp container is

11:57:09.224234 IP 10.110.1.130.45777 > 10.10.3.182.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.35 > 10.0.40.100: ICMP echo request, id 99, seq 98, length 64
11:57:09.232460 IP 10.110.1.129.53901 > 10.110.1.130.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.100 > 10.0.40.35: ICMP echo reply, id 99, seq 98, length 64

I don’t know why the icmp packet doesnt return to the container whereas I saw it on the worker host …

Default iptables Rules:

sudo iptables -L -v -t nat
Chain PREROUTING (policy ACCEPT 4182 packets, 437K bytes)
 pkts bytes target     prot opt in     out     source               destination         
97743 9383K DOCKER-INGRESS  all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
 3930  405K DOCKER     all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 3982 packets, 427K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 1004 packets, 69268 bytes)
 pkts bytes target     prot opt in     out     source               destination         
 2178  175K DOCKER-INGRESS  all  --  any    any     anywhere             anywhere             ADDRTYPE match dst-type LOCAL
    0     0 DOCKER     all  --  any    any     anywhere            !localhost/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 1004 packets, 69268 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 MASQUERADE  all  --  any    !docker0  172.18.0.0/20        anywhere            
  902 45348 MASQUERADE  all  --  any    !docker_gwbridge  172.18.48.0/20       anywhere            
    6   304 MASQUERADE  all  --  any    docker_gwbridge  anywhere             anywhere             ADDRTYPE match src-type LOCAL

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 RETURN     all  --  docker0 any     anywhere             anywhere            
    2   144 RETURN     all  --  docker_gwbridge any     anywhere             anywhere            

Chain DOCKER-INGRESS (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    0     0 DNAT       tcp  --  any    any     anywhere             anywhere             tcp dpt:8082 to:172.18.48.2:8082
    0     0 DNAT       tcp  --  any    any     anywhere             anywhere             tcp dpt:8083 to:172.18.48.2:8083
    0     0 DNAT       tcp  --  any    any     anywhere             anywhere             tcp dpt:http-alt to:172.18.48.2:8080
    0     0 DNAT       tcp  --  any    any     anywhere             anywhere             tcp dpt:9090 to:172.18.48.2:9090
 3965  409K RETURN     all  --  any    any     anywhere             anywhere            
docker network inspect test-arsenal 
[
    {
        "Name": "test",
        "Id": "8nopak0wwpxlb75gg9y8mrtcq",
        "Created": "2022-05-10T14:17:57.193769059+02:00",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.40.0.0/16",
                    "Gateway": "10.40.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "1a94a2c6937a8bb8c3d8d2bf59da0d16ed48e604217185f827b9bf14b1976bd1": {
                "Name": "test-zap-1.1.2o9zf96c62emx4bjyflgmuau7",
                "EndpointID": "0cfafef8f2f318b31b91f4b5ecc59460688d2c952a2e428dff68c9f648480d3e",
                "MacAddress": "02:42:0a:28:00:62",
                "IPv4Address": "10.40.0.98/16",
                "IPv6Address": ""
            },
            "315c71b8c571cf430d8c061b51ae731b1e9d9196518998b0f65cf4086140ff4a": {
                "Name": "test-nessus-1.1.p3mpildppvby7x77a5zj0a0tr",
                "EndpointID": "23156f996b5cf88763590bcea64941fbe64fef2feee507a8e9b2d6b6e698fa95",
                "MacAddress": "02:42:0a:28:00:13",
                "IPv4Address": "10.40.0.19/16",
                "IPv6Address": ""
            },
            "53154c800f7929037464607fbfd9b1b9368f4f740e8f2bc8c4ddc628b6b83900": {
                "Name": "test-web.1.ag35t66u9ojet8wamllkpicvt",
                "EndpointID": "b85ddd8c5c3f3ed4dd7e92dd7fffe69ee0a977ec1e1ff62dd32c4b5f8117e192",
                "MacAddress": "02:42:0a:28:00:64",
                "IPv4Address": "10.40.0.100/16",
                "IPv6Address": ""
            },
            "7cf3fa6eff6031b6a1c5dc2d7a68244265e9a6db70588050576a0f09c810e7a2": {
                "Name": "test-rabbitmq.1.3khyzd4fglcyirq6ymkovts71",
                "EndpointID": "f464be9d41bf8d875d5f21458a2bf9f07e8d90a067e22fe40deefba2c304a477",
                "MacAddress": "02:42:0a:28:00:69",
                "IPv4Address": "10.40.0.105/16",
                "IPv6Address": ""
            },
            "809f9c10fddc331b530574016f475157642b60e57502a21fe03116a4652e5e16": {
                "Name": "test-ssllabs-1.1.6tiat5d69u8pf3hmt4kbiwee3",
                "EndpointID": "77e5c503a5978937b66fa9bb4ed9fa3bd7c81e39aca3c11b8e53e4c9b47c4cff",
                "MacAddress": "02:42:0a:28:00:1c",
                "IPv4Address": "10.40.0.28/16",
                "IPv6Address": ""
            },
            "850abc0fd42f5f12455837c9f9e4719b8b350acef4dc5694ae51e93f753f2724": {
                "Name": "test-wpscan-1.1.q3t2nyv0mlyry274n5jd1ywb6",
                "EndpointID": "fa471b40b95191f45d05f8e9b8e8d206627c086f670ccaf0a5aa5ec7555ffc33",
                "MacAddress": "02:42:0a:28:00:38",
                "IPv4Address": "10.40.0.56/16",
                "IPv6Address": ""
            },
            "lb-test-arsenal": {
                "Name": "test-endpoint",
                "EndpointID": "cbe916c35f8ccde5cc8c405cc973fd9688bafb25056494139c00b0ff86a00864",
                "MacAddress": "02:42:0a:28:00:14",
                "IPv4Address": "10.40.0.20/16",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4148"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "86cfc7ed5320",
                "IP": "10.110.1.130"
            },
            {
                "Name": "39230bb49658",
                "IP": "10.10.2.102"
            },
            {
                "Name": "848bb0c1d6fb",
                "IP": "10.10.3.182"
            },
            {
                "Name": "af036cd6708e",
                "IP": "10.10.3.26"
            }
        ]
    }
]

Thanks for your time