Where can I see relevant logs when such failures occur? We are trying to add multiple overlay networks to get around the /24 default subnet limit, whats the best practice on increasing the number of ips in a cluster. We need to create a cluster where a central orchestrator needs to talk to 1000s of containers, can we increase the subnet size to say /16 or is there any other best practices on how to do this.
Also are there any limitations on how many overlay networks a container can be part of? To work around the /24 limit (https://docs.docker.com/v17.09/engine/swarm/networking/#overlay-network-size-limitations) we are planning to make our orchestrator container join multiple overlay networks and have many endpoint containers per overlay network. Orchestrator needs to connect to 1000s of endpoints.
This blogpost might help you:
https://blog.codeship.com/running-1000-containers-in-docker-swarm/
Also, it seems there is a limit with the number of services run in vip mode (which is the default) - as far as I know, the restriction does not apply “mode: dnsrr” in combination with a custom network.