When multiple containers are launched simultaneously, the startup time for containers increases with the number of containers being launched. For example, if I try to launch 70 containers, it takes ~20 seconds for all containers to be running. After some performance debugging using pprof and perf, it was observed that docker spends majority its time connecting each container to the network (possibly spinning on iptable lock). If I disable the network (–network=none), the startup time for 70 containers is ~8 seconds.
Is this an expected behavior? Does docker not scale well due to network bottleneck? Is there a workaround?
OS Version: Ubuntu 16.04.1 LTS
Docker version: Docker version 17.07.0-dev, build
Steps to reproduce:
Save the current time
Start 70 containers simultaneously (container need not do any operation)
Measure the startup time for each container using the docker inspect command and the saved time
Repeat the steps by setting the network to “none” (–network=none)
I am observing the same issue. With multiple containers the start up time grows exponentially when started with a network. Anyone have a solution or possible reason?
When launching a large number of Docker containers simultaneously, such as 70, you may experience increased startup times due to network configuration overhead. This delay is primarily caused by Docker’s process of connecting each container to the network, which involves updating iptables rules. This network-related bottleneck can lead to significant contention and increased startup times. For example, while launching containers with network connectivity can take around 20 seconds, disabling the network (using --network=none) can reduce this time to approximately 8 seconds. This behavior suggests that Docker’s performance in scaling container launches may be hindered by network configuration tasks. To mitigate this, you might consider optimizing Docker networking settings, batching container launches, tuning iptables performance, updating Docker and your operating system, or using container orchestration solutions like Docker Swarm or Kubernetes for more efficient management. [mod update: remove link]