Container cannot ping another container on a diff host with consul

Hello,
I have 3 CentoS 7 machines.
I tried to set up a cassandra cluster.
cassandra node can discover all other nodes if they are in the same network, therefore i was looking for a way to set up 2 containers on different hosts but still make them work as if they are in the same network

i tried doing it with consul

the steps i did:

  1. creating a consul container on machine0
    docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server -bootstrap
  2. adding the consul configuration to /lib/systemd/system/docker.service in machine1 and machine2
    -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --cluster-store=consul://machine0-external-ip:8500/network --cluster-advertise=machine0-external-ip:2375 --userland-proxy=false
    3.creating an overlay network
    docker network create -d overlay --subnet=10.10.10.0/24 testNetwork
    4.creating busybox containers on both machine1 and machine2, with my test network
    machine1-
    docker run -itd --name container1 --net testNetwork busybox
    machine2-
    docker run -itd --name container2 --net testNetwork busybox
  3. from either machine (1 or 2 ) i could see the containers in the network through this command
    docker network inspect testNetwork
  4. trying to ping container2 from inside container1
    docker exec container1 ping -w 3 container2

this results in
PING container2 (10.10.10.3): 56 data bytes

— container2 ping statistics —
3 packets transmitted, 0 packets received, 100% packet loss

you can see the ping is going to the subnet address ( 10.10.10.3) which is what i want.

What am i doing wrong?
To my understanding the container suppose to go through the autocreated network “docker_gwbridge” in order to know where does this subnet is for real ( which is on a different machine) i ran a sniffer on the packets and it seems like the container just looks up that address in his inner network and doesnt even go through the docker_gwbridge…

if anyone got a different approach\ suggestions to my problem i would love to hear it as well!

Thanks in advance!