I have setup 2 docker swarm nodes on 2 VMs, both of them are manager.
However the first VM, called NAT VM, is using NAT interface with proper port forwarding rules.
The second is using bridge interface and the swarm is created by this node.
docker swarm init --advertise-addr
The swam join was successful on NAT VM.
docker swarm join --advertise-addr --listen-addr
All nodes are reachable and active as seen by “docker node ls”
The test services are running on both nodes.
The problem occurs when I tried to ping from the container running in NAT node to the container on the other end.
Here is the result from tcpdump -i enps03 port 4789 on NAT VM. Noticed the “echo replies” were received.
16:20:16.415449 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097
IP 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 4746, length 64
16:20:16.417093 IP _gateway.43645 > nat_node.4789: VXLAN, flags [I] (0x08), vni 4097
IP 10.0.0.3 > 10.0.0.6: ICMP echo reply, id 72, seq 4746, length 64
16:20:17.417299 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097
However when I enter the network namespace and do tcpdump, I only see the “echo requests” but not the “echo replies”
16:26:27.043329 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5116, length 64
16:26:28.045229 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5117, length 64
16:26:29.047107 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5118, length 64
I think there could be something wrong with my configuration. I am not sure how to do further investigation on this issue.
Debug Mode: false
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries s
Is Manager: true
Default Address Pool: 10.0.0.0/8
Data Path Port: 4789
Task History Retention Limit: 5
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Heartbeat Period: 5 seconds
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.109
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Kernel Version: 4.15.0-58-generic
Operating System: Ubuntu 18.04.3 LTS
Total Memory: 985.5MiB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Live Restore Enabled: false