Docker Swarm custom network doesn't work across servers, while ingress does

Docker Engine Swarm
,
1

Basically, the title. I pushed my setup for swarm here - GitHub - sssemil/docker_swarm_test

I’ve been struggling with this issue for a while now, and I can’t find any solution… so any insight is greatly appreciated. The test setup is simple, two Ubuntu 22.04 server VMs with docker 24.0.5 setup - one is the manager and one is the worker.

Here’s the compose file:

version: '3.7'
services:

  test1:
    image: alpine:latest
    command: nc -l -k 4441
    ports:
      - "4441:4441"
    deploy:
      placement:
        constraints:
          - node.labels.test1 == true
    networks:
      - test-network

  test2:
    image: alpine:latest
    command: nc -l -k 4442
    ports:
      - "4442:4442"
    deploy:
      placement:
        constraints:
          - node.labels.test2 == true
    networks:
      - test-network

networks:
  test-network:
    driver: overlay
    attachable: true
    driver_opts:
      encrypted: "false"

Here’s the manager:

ubuntu@manager:~$ docker ps
CONTAINER ID   IMAGE           COMMAND           CREATED         STATUS         PORTS     NAMES
a0b9f5693271   alpine:latest   "nc -l -k 4441"   6 seconds ago   Up 4 seconds             test_test1.1.mu12mj1om0y58hprewprjo66i
ubuntu@manager:~$ docker exec -it test_test1.1.mu12mj1om0y58hprewprjo66i ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:00:00:07 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.7/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
15: eth2@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth2
       valid_lft forever preferred_lft forever
17: eth1@if18: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1424 qdisc noqueue state UP
    link/ether 02:42:0a:00:01:06 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.6/24 brd 10.0.1.255 scope global eth1
       valid_lft forever preferred_lft forever
ubuntu@manager:~$ docker exec -it test_test1.1.mu12mj1om0y58hprewprjo66i ping 10.0.1.3
PING 10.0.1.3 (10.0.1.3): 56 data bytes
^C
--- 10.0.1.3 ping statistics ---
4 packets transmitted, 0 packets received, 100% packet loss
ubuntu@manager:~$ docker exec -it test_test1.1.mu12mj1om0y58hprewprjo66i ping 10.0.0.5
PING 10.0.0.5 (10.0.0.5): 56 data bytes
64 bytes from 10.0.0.5: seq=0 ttl=64 time=4.283 ms
64 bytes from 10.0.0.5: seq=1 ttl=64 time=3.006 ms
64 bytes from 10.0.0.5: seq=2 ttl=64 time=0.412 ms
^C
--- 10.0.0.5 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.412/2.567/4.283 ms
ubuntu@manager:~$ docker network inspect test_test-network
[
    {
        "Name": "test_test-network",
        "Id": "ygzlx638oukx4ydx8vlbvufep",
        "Created": "2023-11-10T07:29:59.689264454Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "a0b9f56932710beb135a7b0733520c5eeeaff25f9a6391c3b54cadf6b7587c2d": {
                "Name": "test_test1.1.mu12mj1om0y58hprewprjo66i",
                "EndpointID": "0aff4caca4a1062ef2b19bac051ca7349d8fd39873612a9476f2619df54be62a",
                "MacAddress": "02:42:0a:00:01:06",
                "IPv4Address": "10.0.1.6/24",
                "IPv6Address": ""
            },
            "lb-test_test-network": {
                "Name": "test_test-network-endpoint",
                "EndpointID": "d93ed026442f2a1bfa3edc8e1584f933a17807147f09ff6bb11af6ee1fceb463",
                "MacAddress": "02:42:0a:00:01:07",
                "IPv4Address": "10.0.1.7/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "encrypted": "false"
        },
        "Labels": {
            "com.docker.stack.namespace": "test"
        },
        "Peers": [
            {
                "Name": "2174400df87b",
                "IP": "192.168.122.9"
            },
            {
                "Name": "9cc8807b40e9",
                "IP": "192.168.122.38"
            }
        ]
    }
]
ubuntu@manager:~$ docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "i740z3miik87stebxa1ak7s2b",
        "Created": "2023-11-10T07:29:35.571958709Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "a0b9f56932710beb135a7b0733520c5eeeaff25f9a6391c3b54cadf6b7587c2d": {
                "Name": "test_test1.1.mu12mj1om0y58hprewprjo66i",
                "EndpointID": "6fea7cc93050d77d038e38cae1381bff999a5967d0c67ba42cabdbc8afafa3fc",
                "MacAddress": "02:42:0a:00:00:07",
                "IPv4Address": "10.0.0.7/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "e01226fed852ce74367317f8eb00b5eb17aaefdca605ae9b29470660c9c5e515",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "9cc8807b40e9",
                "IP": "192.168.122.38"
            },
            {
                "Name": "2174400df87b",
                "IP": "192.168.122.9"
            }
        ]
    }
]

And here’s the worker:

ubuntu@worker:~$ docker ps
CONTAINER ID   IMAGE           COMMAND           CREATED          STATUS          PORTS     NAMES
2eea0ad6fb83   alpine:latest   "nc -l -k 4442"   11 seconds ago   Up 10 seconds             test_test2.1.rth0nhp5pu0ptaw8y3i6xouio
ubuntu@worker:~$ docker exec -it test_test2.1.rth0nhp5pu0ptaw8y3i6xouio ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
13: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1450 qdisc noqueue state UP
    link/ether 02:42:0a:00:00:05 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.5/24 brd 10.0.0.255 scope global eth0
       valid_lft forever preferred_lft forever
15: eth2@if16: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 brd 172.18.255.255 scope global eth2
       valid_lft forever preferred_lft forever
17: eth1@if18: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1424 qdisc noqueue state UP
    link/ether 02:42:0a:00:01:03 brd ff:ff:ff:ff:ff:ff
    inet 10.0.1.3/24 brd 10.0.1.255 scope global eth1
       valid_lft forever preferred_lft forever
ubuntu@worker:~$ docker network inspect test_test-network
[
    {
        "Name": "test_test-network",
        "Id": "ygzlx638oukx4ydx8vlbvufep",
        "Created": "2023-11-10T07:29:58.00883498Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": true,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2eea0ad6fb8388ca99bb670f1b0ee6434af1d63054ee6ac0a2844c242768902a": {
                "Name": "test_test2.1.rth0nhp5pu0ptaw8y3i6xouio",
                "EndpointID": "c0679573e27c9bdb0b05481d44c2a936bc5f7518296a74b6057b95ee01fd6d2e",
                "MacAddress": "02:42:0a:00:01:03",
                "IPv4Address": "10.0.1.3/24",
                "IPv6Address": ""
            },
            "lb-test_test-network": {
                "Name": "test_test-network-endpoint",
                "EndpointID": "d126d3fec01435d4a660bfe5743308592e75fc9f2982a3a159f64ddebce9ca2e",
                "MacAddress": "02:42:0a:00:01:04",
                "IPv4Address": "10.0.1.4/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097",
            "encrypted": "false"
        },
        "Labels": {
            "com.docker.stack.namespace": "test"
        },
        "Peers": [
            {
                "Name": "2174400df87b",
                "IP": "192.168.122.9"
            },
            {
                "Name": "9cc8807b40e9",
                "IP": "192.168.122.38"
            }
        ]
    }
]
ubuntu@worker:~$ docker network inspect ingress
[
    {
        "Name": "ingress",
        "Id": "i740z3miik87stebxa1ak7s2b",
        "Created": "2023-11-10T07:29:44.436539318Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": true,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "2eea0ad6fb8388ca99bb670f1b0ee6434af1d63054ee6ac0a2844c242768902a": {
                "Name": "test_test2.1.rth0nhp5pu0ptaw8y3i6xouio",
                "EndpointID": "a09d1f67bbaf0d164cfd469a686696b52e014ddaf74ffbe3b1d2e80e07294ca4",
                "MacAddress": "02:42:0a:00:00:05",
                "IPv4Address": "10.0.0.5/24",
                "IPv6Address": ""
            },
            "ingress-sbox": {
                "Name": "ingress-endpoint",
                "EndpointID": "841a25f3dd5b725b625951d530c320fdc1b3feaf33cda0dcc6f8f2bd2681c981",
                "MacAddress": "02:42:0a:00:00:03",
                "IPv4Address": "10.0.0.3/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "9cc8807b40e9",
                "IP": "192.168.122.38"
            },
            {
                "Name": "2174400df87b",
                "IP": "192.168.122.9"
            }
        ]
    }
]

As you can see, ping only works with ingress IPs, but not over test-network IPs.

2

Wow, I checked your setup, seems like a lot a firewall rule bending. We run Swarm without any if that.

What does docker node ls tell you?

3

hey @bluepuma77, yeah haha, otherwise ufw doesn’t block docker ports at all, got that from here - GitHub - chaifeng/ufw-docker: To fix the Docker and UFW security flaw without disabling iptables

here’s docker node ls output:

ID                            HOSTNAME   STATUS    AVAILABILITY   MANAGER STATUS   ENGINE VERSION
kqs4to52p9u21nqeu6flx3gst *   manager    Ready     Active         Leader           24.0.5
wew35yrwmr2h2jki35tv2sdqc     worker     Ready     Active                          24.0.5
4

How do you deploy the compose file?

5

Which vm platform are you using?

6

I use a script I wrote, but basically, after copying it to the manager server, like this: ssh ubuntu@192.168.122.38 'docker stack deploy --with-registry-auth --compose-file ~/docker/compose.yml test where 192.168.122.38 is manager’s IP.

7

I use libvirt for my vm platform

8

made me think about this, and I disabled ufw on the worker, and it pings now, interesting that ingress was still working; also interesting that even after enabling ufw back, ping still works.

In dmesg I see:

[  194.296060] [UFW BLOCK 1] IN=enp1s0 OUT= MAC=52:54:00:3d:c0:ea:52:54:00:55:c0:21:08:00 SRC=192.168.122.38 DST=192.168.122.9 LEN=136 TOS=0x00 PREC=0x00 TTL=64 ID=1269 PROTO=ESP SPI=0xc557d24d

UFW BLOCK 1 because I changed /etc/ufw/user.rules to have:

-A ufw-after-logging-input -j LOG --log-prefix "[UFW BLOCK 1] "
-A ufw-after-logging-forward -j LOG --log-prefix "[UFW BLOCK 2] "
-A ufw-logging-deny -j LOG --log-prefix "[UFW BLOCK 3] "

So, I need to allow this ESP protocol, which is IPsec I think.

Looks like this does exactly that:

sudo ufw allow in on enp1s0 proto esp

And it pings now :slight_smile:

1 Like