Auto network scaling inside of overlay

General Discussions
1

Hi there,

I have been playing with the custom swarm overlays as im needing to deploy a service that will fluctuate between 30 containers up to around 500 containers on the 1 overlay.

I have been using the getting started example from the site to play with this and currently I can have it scale up to 248 containers but it wont scale any more, im sure this is just a simple command or piece of config which I am missing so any suggestions or help would be greatly appreciated as im stuck and not sure where to go from here.

version: "3.7"

services:
  app:
    image: 120gbssd/getting-started
    ports:
      - 3000-3050:3000
    deploy:
      replicas: 350
    networks:
      - getting-started-db-network
      - getting-started-app-network

    environment:
      MYSQL_HOST: mysql
      MYSQL_USER: root
      MYSQL_PASSWORD: secret
      MYSQL_DB: todos

  mysql:
    image: mysql:5.7
    networks:
      - getting-started-db-network
    volumes:
      - todo-mysql-data:/var/lib/mysql
    environment:
      MYSQL_ROOT_PASSWORD: secret
      MYSQL_DATABASE: todos
      MAX_CONNECTIONS: 10000

volumes:
  todo-mysql-data:

networks:
  getting-started-app-network:
    external: true
  getting-started-db-network:
    external: true

getting-started-app-network overlay

[
    {
        "Name": "getting-started-app-network",
        "Id": "w20w40yhj16ha5uc85m4g9v2f",
        "Created": "2022-08-24T07:02:28.346747953Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.51.0.0/16",
                    "Gateway": "10.51.0.1"
                }
            ]
        },
        "Internal": true,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": null
    }
]

getting-started-db-network overlay

[
    {
        "Name": "getting-started-db-network",
        "Id": "pv52s8xiy19qvmdhlci3qzyfg",
        "Created": "2022-08-24T07:02:18.573044826Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.50.0.0/16",
                    "Gateway": "10.50.0.1"
                }
            ]
        },
        "Internal": true,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4097"
        },
        "Labels": null
    }
]
2

Sorry I totally missed this piece so im adding it in late :confused:

  • Issue type
    Docker doesnt autoscale subnets once it hits 254 containers on a single subnet even though I have configured a /16 subnet.

  • OS Version/build
    Host machine: Ubuntu 20.04.4 LTS (GNU/Linux 5.4.0-124-generic x86_64)

  • App version
    Docker version 20.10.17, build 100c701

  • Steps to reproduce
    configure a /16 overlay and then try to launch 300 containers into the one overlay.

3

I don’t recall, wether there is a way arround the limitation or not. But since peolpe tested it with more than 300 containers, I’d assume either the network cidr range should make it possible (which your network inspect inidcates you already use, so this can’t be it alone) and/or some settings in the daemon.json.

I found this blog post, which seem to takle the issue as well: Running 1,000 Containers in Docker Swarm | Cloudbees Blog

4

Hi @meyay thanks for that and for the links as well this gives me a great foot to get started on as I have been pretty stuck on it, I will have a read of the links you have provided and have see if I can get it to work :slight_smile:

Appreciate the help!