I have setup 2 docker swarm nodes on 2 VMs, both of them are manager.
However the first VM, called NAT VM, is using NAT interface with proper port forwarding rules.
The second is using bridge interface and the swarm is created by this node.
docker swarm init --advertise-addr
The swam join was successful on NAT VM.
docker swarm join --advertise-addr --listen-addr
All nodes are reachable and active as seen by “docker node ls”
The test services are running on both nodes.
The problem occurs when I tried to ping from the container running in NAT node to the container on the other end.
Here is the result from tcpdump -i enps03 port 4789 on NAT VM. Noticed the “echo replies” were received.
16:20:16.415449 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097
IP 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 4746, length 64
16:20:16.417093 IP _gateway.43645 > nat_node.4789: VXLAN, flags [I] (0x08), vni 4097
IP 10.0.0.3 > 10.0.0.6: ICMP echo reply, id 72, seq 4746, length 64
16:20:17.417299 IP nat_node.58620 > 192.168.1.108.4789: VXLAN, flags [I] (0x08), vni 4097
However when I enter the network namespace and do tcpdump, I only see the “echo requests” but not the “echo replies”
16:26:27.043329 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5116, length 64
16:26:28.045229 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5117, length 64
16:26:29.047107 02:42:0a:00:00:06 > 02:42:0a:00:00:03, ethertype IPv4 (0x0800), length 98: 10.0.0.6 > 10.0.0.3: ICMP echo request, id 72, seq 5118, length 64
I think there could be something wrong with my configuration. I am not sure how to do further investigation on this issue.
Thank you.
Client:
Debug Mode: false
Server:
Containers: 2
Running: 2
Paused: 0
Stopped: 0
Images: 3
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries s
Swarm: active
NodeID: m0kkg71w5p3z25vg090g380j1
Is Manager: true
ClusterID: w1g7axnt3sv9mniimcooh51h1
Managers: 2
Nodes: 2
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Data Path Port: 4789
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 192.168.1.109
Manager Addresses:
192.168.1.108:2377
192.168.1.109:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.0-58-generic
Operating System: Ubuntu 18.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 985.5MiB
Name: nat_node
ID: GHF5:I7RQ:UX4Z:Q3NM:SKO7:XJUX:H2XW:O6IB:KHSG:PKBM:NCGX:WPAR
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false