Cannot ping containers of same private network deployed across multiple hosts

General
1

I have deployed overlay network to connect 2 private networks having 4 containers each.
Each private network has 1 fog and 3 edge devices.
Each private network is clique.

I am not able to ping from container A on host X to container B on host Y (across hosts). What could be the reason?

created consul on Admin hosts

docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap

#started docker on hosts used to deploy setup
nohup /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-advertise :2375 --cluster-store consul://:8500 &

image used to spin container

docker pull shrey67/centos_systemd

create overlay

udo docker network create -d overlay

create fog

udo docker run --ulimit nofile=50000:50000 -i -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cpus={1} --privileged --cap-add=NET_ADMIN --cap-add=NET_RAW --hostname {0} --name {0} {2} > /dev/null &

create edge

sudo docker run --ulimit nofile=50000:50000 -i -v /sys/fs/cgroup:/sys/fs/cgroup:ro --cpus={1} --privileged --cap-add=NET_ADMIN --cap-add=NET_RAW --hostname {0} --name {0} {2} > /dev/null &

create private network

for fog

“sudo docker network connect {0} {1}”.format(private_network[i],gw)

“sudo docker exec -i {0} service iptables start”.format(gw),“sudo docker exec -i {0} iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE”.format(gw),
“sudo docker exec -i {0} iptables -A FORWARD -i eth0 -o eth1 -m state --state RELATED,ESTABLISHED -j ACCEPT”.format(gw),
“sudo docker exec -i {0} iptables -A FORWARD -i eth1 -o eth0 -j ACCEPT”.format(gw),“sudo docker exec -i {0} iptables -D INPUT -j REJECT --reject-with icmp-host-prohibited”.format(gw),
“sudo docker exec -i {0} iptables -D FORWARD -j REJECT --reject-with icmp-host-prohibited”.format(gw)

#for edge
“sudo docker network connect {0} {1}”.format(private_network[i],device)

“sudo docker network disconnect bridge {0}”.format(device)
“sudo docker network disconnect docker_gwbridge {0}”.format(device)
“sudo docker exec -i {0} tc qdisc add dev eth1 handle 1: root htb default 11”.format(device),
“sudo docker exec -i {0} tc class add dev eth1 parent 1: classid 1:1 htb rate {1}Mbps”.format(device, private_networks_dict[private_network[i]][“bw”]),
“sudo docker exec -i {0} tc class add dev eth1 parent 1:1 classid 1:11 htb rate {1}Mbit”.format(device,private_networks_dict[private_network[i]][“bw”]),
“sudo docker exec -i {0} tc qdisc add dev eth1 parent 1:11 handle 10: netem delay {1}ms”.format(device,int( private_networks_dict[private_network[i]][“latency”]))

“sudo docker exec -i {0} ip route | grep default | awk ‘{{print $3}}’”.format(device)

“sudo docker exec -i {0} route del default gw {1}”.format(device,def_gw)

“sudo docker exec -i {0} route add default gw {1}”.format(device,gw_ip)

“sudo docker exec -i {0} ip a”.format(device)
“sudo docker exec -i {0} ip route”.format(device)

public network

“sudo docker network connect {} {}”.format(public_network[i],device)

Am I missing any configuration?
This configuration works on AWS but not working on Local cluster
Can anyone help me?

source code available on github:

Thanks