I try to do the same setup with wireguard and two cloud provider. All route are ok, I create my cluster and swarm can communicate through UDP and TCP through internal LAN.
Sam problem when services try to communicate on the same overlay networks, the dns docker resolve the ip but the ping doesnt reach the target.
From container with rabbitmq service
ping tasks.rabbitmq
PING tasks.rabbitmq (10.40.0.105) 56(84) bytes of data.
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=1 ttl=64 time=0.219 ms
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=2 ttl=64 time=0.061 ms
64 bytes from patrowl-arsenal_rabbitmq.1.3khyzd4fglcyirq6ymkovts71.patrowl-arsenal (10.40.0.105): icmp_seq=3 ttl=64 time=0.076 ms
^C
--- tasks.rabbitmq ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2022ms
rtt min/avg/max/mdev = 0.061/0.118/0.219/0.071 ms
From the container on the other network through wireguard
ping tasks.rabbitmq
PING tasks.rabbitmq (10.40.0.105): 56 data bytes
^C
--- tasks.rabbitmq ping statistics ---
10 packets transmitted, 0 packets received, 100% packet loss
I have debug a little further with tcpdump and udp port 4789. Because swarm use vxlan techno to send ip packet to another host, I saw the echo request and echo reply in UDP packet:
CF below
10.110.1.130 is the worker where the sender icmp container is
10.10.3.182 is the worker where the target container is
sudo tcpdump -i any -n -p udp port 4789
13:55:38.193961 IP 10.110.1.130.45777 > 10.10.3.182.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.35 > 10.0.40.100: ICMP echo request, id 99, seq 7, length 64
13:55:38.194083 IP 10.10.3.182.53901 > 10.110.1.130.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.100 > 10.0.40.35: ICMP echo reply, id 99, seq 7, length 64
And after on the worker where the sender icmp container is
11:57:09.224234 IP 10.110.1.130.45777 > 10.10.3.182.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.35 > 10.0.40.100: ICMP echo request, id 99, seq 98, length 64
11:57:09.232460 IP 10.110.1.129.53901 > 10.110.1.130.4789: VXLAN, flags [I] (0x08), vni 4120
IP 10.0.40.100 > 10.0.40.35: ICMP echo reply, id 99, seq 98, length 64
I don’t know why the icmp packet doesnt return to the container whereas I saw it on the worker host …
Default iptables Rules:
sudo iptables -L -v -t nat
Chain PREROUTING (policy ACCEPT 4182 packets, 437K bytes)
pkts bytes target prot opt in out source destination
97743 9383K DOCKER-INGRESS all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
3930 405K DOCKER all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 3982 packets, 427K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1004 packets, 69268 bytes)
pkts bytes target prot opt in out source destination
2178 175K DOCKER-INGRESS all -- any any anywhere anywhere ADDRTYPE match dst-type LOCAL
0 0 DOCKER all -- any any anywhere !localhost/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 1004 packets, 69268 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any !docker0 172.18.0.0/20 anywhere
902 45348 MASQUERADE all -- any !docker_gwbridge 172.18.48.0/20 anywhere
6 304 MASQUERADE all -- any docker_gwbridge anywhere anywhere ADDRTYPE match src-type LOCAL
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all -- docker0 any anywhere anywhere
2 144 RETURN all -- docker_gwbridge any anywhere anywhere
Chain DOCKER-INGRESS (2 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:8082 to:172.18.48.2:8082
0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:8083 to:172.18.48.2:8083
0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:http-alt to:172.18.48.2:8080
0 0 DNAT tcp -- any any anywhere anywhere tcp dpt:9090 to:172.18.48.2:9090
3965 409K RETURN all -- any any anywhere anywhere
docker network inspect test-arsenal
[
{
"Name": "test",
"Id": "8nopak0wwpxlb75gg9y8mrtcq",
"Created": "2022-05-10T14:17:57.193769059+02:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.40.0.0/16",
"Gateway": "10.40.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"1a94a2c6937a8bb8c3d8d2bf59da0d16ed48e604217185f827b9bf14b1976bd1": {
"Name": "test-zap-1.1.2o9zf96c62emx4bjyflgmuau7",
"EndpointID": "0cfafef8f2f318b31b91f4b5ecc59460688d2c952a2e428dff68c9f648480d3e",
"MacAddress": "02:42:0a:28:00:62",
"IPv4Address": "10.40.0.98/16",
"IPv6Address": ""
},
"315c71b8c571cf430d8c061b51ae731b1e9d9196518998b0f65cf4086140ff4a": {
"Name": "test-nessus-1.1.p3mpildppvby7x77a5zj0a0tr",
"EndpointID": "23156f996b5cf88763590bcea64941fbe64fef2feee507a8e9b2d6b6e698fa95",
"MacAddress": "02:42:0a:28:00:13",
"IPv4Address": "10.40.0.19/16",
"IPv6Address": ""
},
"53154c800f7929037464607fbfd9b1b9368f4f740e8f2bc8c4ddc628b6b83900": {
"Name": "test-web.1.ag35t66u9ojet8wamllkpicvt",
"EndpointID": "b85ddd8c5c3f3ed4dd7e92dd7fffe69ee0a977ec1e1ff62dd32c4b5f8117e192",
"MacAddress": "02:42:0a:28:00:64",
"IPv4Address": "10.40.0.100/16",
"IPv6Address": ""
},
"7cf3fa6eff6031b6a1c5dc2d7a68244265e9a6db70588050576a0f09c810e7a2": {
"Name": "test-rabbitmq.1.3khyzd4fglcyirq6ymkovts71",
"EndpointID": "f464be9d41bf8d875d5f21458a2bf9f07e8d90a067e22fe40deefba2c304a477",
"MacAddress": "02:42:0a:28:00:69",
"IPv4Address": "10.40.0.105/16",
"IPv6Address": ""
},
"809f9c10fddc331b530574016f475157642b60e57502a21fe03116a4652e5e16": {
"Name": "test-ssllabs-1.1.6tiat5d69u8pf3hmt4kbiwee3",
"EndpointID": "77e5c503a5978937b66fa9bb4ed9fa3bd7c81e39aca3c11b8e53e4c9b47c4cff",
"MacAddress": "02:42:0a:28:00:1c",
"IPv4Address": "10.40.0.28/16",
"IPv6Address": ""
},
"850abc0fd42f5f12455837c9f9e4719b8b350acef4dc5694ae51e93f753f2724": {
"Name": "test-wpscan-1.1.q3t2nyv0mlyry274n5jd1ywb6",
"EndpointID": "fa471b40b95191f45d05f8e9b8e8d206627c086f670ccaf0a5aa5ec7555ffc33",
"MacAddress": "02:42:0a:28:00:38",
"IPv4Address": "10.40.0.56/16",
"IPv6Address": ""
},
"lb-test-arsenal": {
"Name": "test-endpoint",
"EndpointID": "cbe916c35f8ccde5cc8c405cc973fd9688bafb25056494139c00b0ff86a00864",
"MacAddress": "02:42:0a:28:00:14",
"IPv4Address": "10.40.0.20/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4148"
},
"Labels": {},
"Peers": [
{
"Name": "86cfc7ed5320",
"IP": "10.110.1.130"
},
{
"Name": "39230bb49658",
"IP": "10.10.2.102"
},
{
"Name": "848bb0c1d6fb",
"IP": "10.10.3.182"
},
{
"Name": "af036cd6708e",
"IP": "10.10.3.26"
}
]
}
]
Thanks for your time