Share and learn in the Docker community.
We have a fairly large docker swarm deployment. Currently using one of the CNI for networking between the containers. We are looking to replace the CNI with docker overlay. Looking for any suggestions on using docker overlay, any thing need to be aware… Hopefully, docker overlay is actively supported by Docker
Any performance characterization available with docker overlay
Docker Overlay Network is part of Docker Swarm, development seems kind of stalled.
We use it, noteworthy is that by default it only creates a /24 network and you might need to set the MTU when using over VLAN/vSwitch.
Thank your for your response. How big is your deployment… Any thoughts on performance numbers for deployments of size 100VMs with 1000 containers
If you consider using encrypted overlay networks, make sure that IP protocol 50 (IPSec ESP) traffic is allowed amongst the nodes.
Whenever I saw a cluster of this size or bigger, it was a Kubernetes cluster. I can’t say anything about how such a huge swarm cluster performs, I have never seen swarm clusters bigger than 15 nodes.
Out of curiosity, which CNI are you currently using, and why are you looking to replace it with docker overlay?
replacing with weave
Thank you. I have some special interest in weave, which is why I asked.
If you mean Weave Net (link), that hasn’t seen a single commit in 1.2 years, I would not recommend to migrate to that.
No one is taking care of security issues and it might just stop working with the next OS release.
Using Swarm is bad enough with the ever slowing development 
Don’t tell anyone yet, but someone is trying to take care of the Weave security issues. Me.
I’ve been maintaining a fork for my own use, which I plan to announce publicly soon. Still, probably not a good idea to use that in production - at least, not yet.
It’s great to have a new maintainer! 
But as an admin I must also take security into consideration. Single person project, supply chain attacks, how to ensure the software is not compromised? So don’t expect a rapid adoption
I have my issues with docker-socket-proxy, which is used to improve security. But who is maintaining it, can you trust them? The project seems to be maintained on Github (link), but latest Docker image is 3 years old (link), might have multiple security issues. So you want to improve security, but might introduce more security issues 
we are planning to migrate from weave!!! not to weave
Any suggestions on what is more appropriate CNI provider for a pure docker environment for a mid sized deployment [upto 100 VMs and 1000 containers at most]
Good luck with that 
We have a fairly large docker based deployment not docker swarm. Apologies for the typo in my original description
Just in case you would still consider weave as an option, take a look at GitHub - rajch/weave: Simple, resilient multi-host containers networking and more.. We’ve started to put out new releases.
This is only for security patches right ? And not any bugs in weave ?
The original thought was only security patches, but now we are fixing bugs too. For example, a CNI compliance error has been corrected, and Docker API minimum version has been bumped up to API 1.24.
We have rather large (> 250 containers) deployments running on docker swarm, using north of 50 VMs in some of our deployments.
We did have issues with the overlay network at that scale. We addressed them by sometimes allowing a /16 instead of a /24 on the bridge network (not the overlay… Going bigger on the overlay network can cause issues… and we had issues :).
We have made some work to divide our overlay networks in logical entities (Management, prometheus monitoring, data streaming, etc)… In our experience, keeping the number of connected containers to a network < 128 (half of a /24), works for stability.
We have now several years of experience running rather large systems using these premices.
I am looking at using encrypted overlay network on some of these links. Anyone has relevant experience doing this, and what is the performance cost I could expect?)