Docker swarm macvlan cannot connect across host/gateway on google cloud (promisc mode on already)

Open Source Projects Swarm
1

Dear All,

Greeting from thailand. We are try to initial lab for swarm mac-vlan following your suggestion on this blog on google cloud by 3 node of swarm like detail below:

==========================================================================
#Compute Farm#
NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS
swarm-mng asia-east1-a n1-standard-1 192.168.99.200 35.194.247.60 RUNNING
swarm-node1 asia-east1-a n1-standard-1 192.168.99.201 35.194.144.119 RUNNING
swarm-node2 asia-east1-a n1-standard-1 192.168.99.202 35.187.158.143 RUNNING

#Create MACVLAN on each local machine#
#swarm-mng#
#ip range: (192.168.99.129 - 192.168.99.158)#
docker network create --config-only --subnet 192.168.99.0/24
–gateway 192.168.99.1 -o parent=ens4
–ip-range 192.168.99.129/27 macvlannet

#swarm-node1#
#ip range: (192.168.99.161 - 192.168.99.190)#
docker network create --config-only --subnet 192.168.99.0/24
–gateway 192.168.99.1 -o parent=ens4
–ip-range 192.168.99.161/27 macvlannet

#swarm-node2#
#ip range: (192.168.99.193 - 192.168.99.222)#
docker network create --config-only --subnet 192.168.99.0/24
–gateway 192.168.99.1 -o parent=ens4
–ip-range 192.168.99.193/27 macvlannet

#Create Global MACVLAN on Swarm#
#swarm-mng#
docker network create -d macvlan --scope swarm --config-from macvlannet swarm-macvlan
docker network ls

#Result#
NETWORK ID NAME DRIVER SCOPE
9ba217a5bb8f bridge bridge local
63a8a77b28df docker_gwbridge bridge local
810ea4ddbf72 host host local
bxch17e0ojmy ingress overlay swarm
a320770f9521 macvlannet null local
4e51be11f04e none null local
xfjw25nw01q2 swarm-macvlan macvlan swarm

After finished to create macvlan on swarm. We try to create 2 simple alpine linux (with small web on port 3000) service for attach with “swarm-macvlan” and check ip address of them as below:

#swarm-mng#
#create simple alpine service#
docker service create -dt --name alpine1
–replicas 1 --network swarm-macvlan
labdocker/alpineweb:latest node hello.js

docker service create -dt --name alpine2
–replicas 1 --network swarm-macvlan
labdocker/alpineweb:latest node hello.js

#check location of service#
docker service ls
ID NAME MODE REPLICAS IMAGE PORTS
oaxk3ro3pet0 alpine2 replicated 1/1 labdocker/alpineweb:latest
qi0c1t6anhtr alpine1 replicated 1/1 labdocker/alpineweb:latest

docker service ps alpine1
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
ml2dgppfj3y9 alpine1.1 labdocker/alpineweb:latest swarm-node1 Running

docker service ps alpine2
ID NAME IMAGE NODE DESIRED STATE CURRENT STATE ERROR PORTS
hpj009ex200q alpine2.1 labdocker/alpineweb:latest swarm-node2 Running

#check ip address on service alpine1#
praparn@swarm-node1:~$ docker inspect $(docker ps -q)| grep IPAddress
"SecondaryIPAddresses": null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.160”,

#check ip address on service alpine2#
docker inspect $(docker ps -q)| grep IPAddress
"SecondaryIPAddresses": null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.192”,

==========================================================================

So we are try to check connect across host via macvlan and inside connecter test ping/curl to other host/gateway in every test case it cannot operate:

#swarm-mng ==> alpine 1 (on swarm-node1), alpine2 (on swarm-node2):
192.168.99.200 ==> 192.168.99.160
192.168.99.200 ==> 192.168.99.192
Result: request timeout / no arp / curl connection timeout

#alpine 1 (on swarm-node1) ==> gateway 192.168.99.100, swarm-mng
192.168.99.160 ==> 192.168.99.1
192.168.99.160 ==> 192.168.99.200
Result: request timeout / no arp

#alpine 2 (on swarm-node2) ==> gateway 192.168.99.100
192.168.99.192 ==> 192.168.99.1
192.168.99.192 ==> 192.168.99.200
Result: request timeout / no arp

#alpine 1 (on swarm-node1) ==> #alpine 2 (on swarm-node2)
192.168.99.160 ==> 192.168.99.192
Result: request timeout / no arp / curl connection timeout

#alpine 2 (on swarm-node2) ==> alpine 1 (on swarm-node1)
192.168.99.192 ==> 192.168.99.160
Result: request timeout / no arp / curl connection timeout

==========================================================================

After this problem we try to test connect of container inside same host by scale service from 1 = 6 on both alpine1,alpine2

#swarm-node1
docker inspect $(docker ps -q)| grep IPAddress
"SecondaryIPAddresses": null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.162”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.163”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.161”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.160”,

#swarm-node2
docker inspect $(docker ps -q)| grep IPAddress
"SecondaryIPAddresses": null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.195”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.193”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.194”,
“SecondaryIPAddresses”: null,
“IPAddress”: “”,
“IPAddress”: “192.168.99.192”,

#Test ping inside host
#192.168.99.192 ==> 192.168.99.195
Result: successful

#192.168.99.162 ==> 192.168.99.161
Result: successful

From result lab above we found connect can establish on same host only and cannot access to outside. Could you please suggestion or any thing we need to configure on iptable host ?

==========================================================================
Remark:
#OS information
#uname -a (all node with same version)
Linux swarm-mng 4.8.0-56-generic #61~16.04.1-Ubuntu SMP Wed Jun 14 11:58:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

#promisc mode (all node)
ip link set dev ens4 promisc on

#docker info (all node with same version)
docker info
Containers: 4
Running: 4
Paused: 0
Stopped: 0
Images: 1
Server Version: 17.06.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog
Swarm: active
NodeID: d1mtchl37k02njhl6fgsv4eai
Is Manager: true
ClusterID: 4qpj9lyh4zj4k6jjv4dxn1b8h
Managers: 1
Nodes: 3
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 3
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Root Rotation In Progress: false
Node Address: 192.168.99.200
Manager Addresses:
192.168.99.200:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 6e23458c129b551d5c9871e5174f6b1b7f6d1170
runc version: 810190ceaa507aa2727d7ae6f4790c76ec150bd2
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.8.0-56-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.613GiB
Name: swarm-mng
ID: LCJX:TLXM:B3GR:RCKE:3HXU:EP3J:RZJY:2K2Y:M7PC:DMDF:3C3P:GXVB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

Br,
Praparn L

2

i have the same problem did you found a solution ?

3

Hi,

My reply is late but posting it because it might help some other reader.

I had the same issue when I was playing in my local environment, for cross host communication the parent interface needs to be in promiscuous mode .

Consider the parent interface to be eth0.

Execute netstat -i | grep eth0 response will be as below.

eth0 1500 0 2799 0 0 0 980 0 0 0 BMPRU

BMPRU : Secret sauce is here, letter P is important which show the interface is in promiscuous mode .

To enable promiscuous mode
ip link set promisc on

Perform same on all nodes in swarm and you should be able to communicate.