Containers running on Multi Host (two VMs on seperate ESX) networking with overlay network driver

Hi team,

I am stuck with below configuration and not able to proceed further, Need your assistance to resolve the following problem.

For our application use case which is distributed and multi-tier, we are exploring Docker with below configuration. I am using Docker 1.12. (latest) version.
I have two seperate ESX hosts, one ESX has single VM running as docker host(say host-1). Another ESX has another VM running with docker host (say host-2). Both of these VMs are running with CentOS 7.x. I have installed docker engine and docker-machine on both of these hosts.

I have successfully configured the external K-V store using virtualbox network driver . I created the virtualbox on one of the VM (host1) using docker-machine create. Also installed the KV store container consul on one of the docker host. However since these VMs(both docker hosts) have static IP assigned and configured in vLAN236, the virtualbox network interface (vboxnet0) always creates the virtual-box and docker-machines using 192.168.x.x. I would like to configure the docker-machine and virtual box with vLAN236 so that I can leverage centralized virtual box with docker machines running on host-1 as well as host-2. The goal I am trying to achieve is,

  1. I would like to make the virtual box as common between host-1 and host-2.
  2. I would like to create a swarm cluster of VMs configutred on host-1 and host-2 to centralized virtual box network.
  3. Create a overlay network to make the swarm to communicate containers across two different hosts.
  4. Make the containers communicate with external world as well.

What I did so far is:

  1. I tried to configure the vboxnet0 interface created by virtualbox, with static IP. However when I tried to create a virtual machine using “docker-machine” with this virtual box, it always revert back to 192.168.x.x IP. This is making me difficult for communication from other docker host (host2).
  2. I tried DHCP server by providing lower range and upper range using “vboxmanage” and tried to modify the vboxnet0 interface, however this also does not help me since virtualbox network driver always assigns the 192.168.x.x IP.
  3. I tried modifying the vboxnet0 interface in “/etc/sysconfig/network-scripts/ifcfg-vboxnet0”, by assigning static IP. However after reboot, the vboxnet0 still gets the IP range 192.168.x.x.

Please advice me with some examples on how this can be made successful communication to take the advantage of containers running on distributed networked cluster of docker hosts.

with Docker 1.12 you do not need an external KV store in swarm mode. you can use the new network feature Macvlan to create your overlay network using the Macvlan driver. Docker Reference Architecture: Designing Scalable, Portable Docker Container Networks