Share and learn in the Docker community.
Hello all. I am deploying some containers (as a service) using Docker Swarms. I want to do my own load balancing of that service. Thus I must not use virtual ip as the endpoint. However my only other option is round-robin DNS. The DNS is helpful because it does expose the individual ip addresses of the containers, however the round-robin aspect of this DNS is creating other problems for me. How can I stop the round-robin behavior?
Thanks!
Let me guess, the container instances of your service hold state that is not shared amongst the instances?
With or without docker, this will cause headaches with loadbalaning in generall, unless you have sticky sessions where requests of a client all always forwarded to the same instance.
Just a silly idea:
If you run a global loadbalancer service that binds a host port on each node, you could get the clients real ip and use sticky sessions inside your loadbalancer container to direct the request of one client always to the same container instance. Of course if this container instance is lost, the session state is lost with it as well. If you have dnsrr for your nodes, you could still have a single entrypoint url and manage the balancing yourself…
The better approach is to actualy make your application stateless OR externalize the state to a session service (which in the simplest case could be a kv store), where you fetch the latest state of the session when the request arrives and update the latest state before the response is delivered.
Does that make sense?
Hi Metin,
I use a type of “sticky sessions” and my client sessions should always be forwarded to the same instance.
Specifically I achieve sticky sessions via consistent hashing. As a result of consistent hashing, scaling up or down does not invalidate the entire hash map (and thus the data depending on that hash map). Only a small percentage of the hash map is invalid in the case of scale up or down.
Docker has assumed that just because I want a hostname to resolve to multiple IP addresses (via DNS lookup and dnsrr), I also want those IP addresses to be shuffled in a round robin fashion each time the hostname is looked up. That assumption is causing me problems. Specifically that assumption is screwing up the consistent hashing algorithm, which depends on the IP addresses in the list AS WELL AS the order of the address.
I can’t go into detail about my reasons for doing this, other than to say I know what I am doing and I have good reasons for doing so. I will not have the load balancing problems that you are trying to warn me about (I am well aware of them). And I do appreciate your warning me.
As I understand Docker, the default exposure of a service is vip (virtual ip). The user can change that to dnsrr. I wand the dns part, but not the rr part. Is there some way to achieve the dns without the rr??
In my basic searching, this is trivial in kubernetes. Im trying to stick to Docker though.
Best,
Ian
I am afraid this is not possible. Would it help if all containers would have different hostnames? If so you could use hostname: ' yourprefix{{ .Task.Slot}}'. Task.Slot will be the number of the replica, thus a predictable number between 1 and your replica count. When one of the replicated containers dies, its slot frees up and is reused when the scheduler creates a new task to summon a new container. You could add each of the individual hostnames as a target backend in your loadbalancer. I know this solution is far away from beeing perfect…
Also you might want to get rid of deploy.mode: ingress for the published port of your loadbalancer and instead use deploy.mode: global in combination with ports.mode: host to actualy get the clients ip in order to hash it properly.
Hi Metin,
It would not help if the containers had different hostnames. This would make matters worse. The reason is that the number of containers is variable. It can go up or down while the cluster is running, but is always greater than 1. I want to avoid having to update the loadbalancer (or its config) because a new container is added or removed.
Yes, I need to make use of host ports. I am already doing that. Ports are actually not the issue. The IP addresses are the issue. Docker wants to return those to me in an unpredictable (or at least changing) order. For most applications that is a feature and a benefit. But not in all cases. In my case I need a non-changing order.
Since we have started our conversation, I have decided to start updating the libnetwork code that Docker relies on for its DNS. This may be the only path forward.
Do you have any other thoughts on ways to achieve sorted DNS results?
Best,
Ian
if you won’t/can’t modify your application, modifiying libnetwork seems like a good approach.
Keep us updated please, I am realy currious how this plays out 
Thank you for your support in this 
The first time I tried to fix a problem (in chmod and chown in coreutils) it didn’t go well… too much bureaucracy… but it was a great learning experience.
So, this time I hope that others can see some benefit in the new features, and that it can get into master branch.
Do you have need/use for such a feature as what I am talking about?
I have posted about it here:
https://github.com/docker/libnetwork/issues/2502
Best,
Ian