Docker Community Forums

Share and learn in the Docker community.

Scale by using overlay network without swarm

Hey guys,

I am trying to scale my current application sitting in a single host to a multi-host architecture with as minimal effort as possible. i.e. not really want to use Swarm or even Kubernetes as seems overkill

I do not really need replica management provided by Swarm or Kubernetes as I think only one container is enough. The use case is to only spin up several more containers in another new server and at the same time, they can talk to some/all containers (ideally by referring to the container name) in the original server.

My idea is to keep everything simple by keeping everything untouched in the current server A running my app. However, add a new server B in the same network to accommodate a few more containers (those containers can be though as stateless worker containers to consume tasks in a queue in the Redis container within server A).

Currently, server A just simply uses a custom bridge network to connect all containers together in the same host, which works fine. Also, I know that I can use an overlay network to achieve across-node communication between containers.

So my questions are:

  1. First, Can I achieve it without Swarm? or still good to use swarm due to some reasons?

  2. Second, for each container currently in server A, do all of them need to replace the current bridge network with a new overlay network? or only part of them that need to talk with containers in server B without changing the bridge network for other containers? or one container can join two networks (i.e. bridge and overlay)

I will answer some parts of you question:

Swarm already encapsulates all the logic to create an overlay network accross nodes.I have never checked how they actualy do it under the hood. You would need to perform all the tasks by yourself and figure out how to make the docker network logic actualy work with it.

Compared to Kubernetes, Swarm realy is a simple orchestrator in terms of ease of use, but also in terms of features. While Learning swarm is like learning how to ride a bike, learning Kubernetes is more like learning how to fly different types of planes ^^ Though, if you require privileged containers, --device, set ulimits, add additional capabilites or want to assign static ips to your containers: up to Docker 19.03.x, Swarm is not able to do any of this. Oh and swarm takes disposable container seriously: on every service start, the containers get re-created. Thus, you need to make sure the persistant data is stored in volumes and not accidently in the cow-layer of the container.

Though, persistent storage can be challenging, as a declaration for volumes is local to nodes. A swarm service will create the declaration on each node a container requiring a volumes is started the first time. You will either need to use volumes backed by a remote share (nfs/cifs) or a docker volume plugin that serves your needs. But, there is a way arround this: to prevent a container to be started on a random node, you can add node labels to your nodes and use placement constraints in your service declarations to “stick them” to specific nodes. If a container always commes up on the same node, there is no necessity for volumes backed by a remote share.

Pretty much depends on what the containers do. A container can participate in one or more network.

Most users never need to configure the ingress network, but Docker 17.05 and higher allow you to do so. This can be useful if the automatically-chosen subnet conflicts with one that already exists on your network, or you need to customize other low-level network settings such as the MTU.

Customizing the ingress network involves removing and recreating it. This is usually done before you create any services in the swarm. If you have existing services which publish ports, those services need to be removed before you can remove the ingress network.

During the time that no ingress network exists, existing services which do not publish ports continue to function but are not load-balanced. This affects services which publish ports, such as a WordPress service which publishes port 80.

Inspect the ingress network using docker network inspect ingress, and remove any services whose containers are connected to it. These are services that publish ports, such as a WordPress service which publishes port 80. If all such services are not stopped, the next step fails.

Remove the existing ingress network:

$ docker network rm ingress

WARNING! Before removing the routing-mesh network, make sure all the nodes
in your swarm run the same docker engine version. Otherwise, removal may not
be effective and functionality of newly created ingress networks will be
impaired.
Are you sure you want to continue? [y/N]
Create a new overlay network using the --ingress flag, along with the custom options you want to set. This example sets the MTU to 1200, sets the subnet to 10.11.0.0/16, and sets the gateway to 10.11.0.2.

$ docker network create
–driver overlay
–ingress
–subnet=10.11.0.0/16
–gateway=10.11.0.2
–opt com.docker.network.driver.mtu=1200
my-ingress
Note: You can name your ingress network something other than ingress, but you can only have one. An attempt to create a second one fails.

Restart the services that you stopped in the first step.