Why Docker's network init is so slow on Linux host (but not on macOS!) and how do I make it faster?

Docker Engine General
1

Hey all,

I’m planning on running a bunch of integration tests with Docker (Compose). Given that every test is going to be run in its own isolated environment, and that there are many of them, I’d like to optimize test startup time as much as I can.

I’ve encountered an issue of docker-compose up being unacceptably slow when run natively on Linux (Ubuntu 18.04) with network_mode: bridge (default networking enabled). Weirdly enough, Docker Desktop for Mac does not have the same issue, and docker-compose up is 2x faster (7 s vs 3.8 s) on a VM that’s run on my Mac laptop! However, with network_mode: none, both services take comparable times to start.

Sample docker-compose.yml:

version: "3"

services:

    test1:
        image: ubuntu:xenial-20190222
        command: pwd

    test2:
        image: ubuntu:xenial-20190222
        stop_signal: SIGKILL
        command: sleep infinity

    test3:
        image: ubuntu:xenial-20190222
        stop_signal: SIGKILL
        command: sleep infinity

    test4:
        image: ubuntu:xenial-20190222
        stop_signal: SIGKILL
        command: sleep infinity

    test5:
        image: ubuntu:xenial-20190222
        stop_signal: SIGKILL
        command: sleep infinity

Test run on Ubuntu 18.04 (7.00 s):

ubuntu$ /usr/bin/time docker-compose up --abort-on-container-exit
Starting ubuntu_test1_1 ... done
Starting ubuntu_test4_1 ... done
Starting ubuntu_test5_1 ... done
Starting ubuntu_test3_1 ... done
Starting ubuntu_test2_1 ... done
Attaching to ubuntu_test4_1, ubuntu_test5_1, ubuntu_test1_1, ubuntu_test3_1, ubuntu_test2_1
test1_1  | /
ubuntu_test1_1 exited with code 0
Aborting on container exit...
Stopping ubuntu_test5_1 ... done
Stopping ubuntu_test3_1 ... done
Stopping ubuntu_test4_1 ... done
Stopping ubuntu_test2_1 ... done
0.56user 0.04system 0:07.00elapsed 8%CPU (0avgtext+0avgdata 41688maxresident)k
0inputs+0outputs (0major+16834minor)pagefaults 0swaps

Test run on macOS 10.14.4 (3.86 s):

macos$ gtime docker-compose up --abort-on-container-exit                                                                                                      
Starting pypt_test1_1 ... done
Starting pypt_test5_1 ... done
Starting pypt_test4_1 ... done
Starting pypt_test2_1 ... done
Starting pypt_test3_1 ... done
Attaching to pypt_test2_1, pypt_test3_1, pypt_test4_1, pypt_test5_1, pypt_test1_1
test1_1  | /
pypt_test1_1 exited with code 0
Aborting on container exit...
Stopping pypt_test5_1 ... done
Stopping pypt_test2_1 ... done
Stopping pypt_test4_1 ... done
Stopping pypt_test3_1 ... done
0.68user 0.19system 0:03.86elapsed 22%CPU (0avgtext+0avgdata 29300maxresident)k
0inputs+0outputs (9major+14831minor)pagefaults 0swaps

Again, if one sets network_mode to none on all containers in docker-compose.yml, startup time gets reduced to ~2 s on both Ubuntu and macOS.

I’ve tried looking into Docker daemon’s log (relevant dockerd.log from Ubuntu test run: https://pastebin.com/raw/XAk3WF20), and I think it’s the network that takes forever to get set up on Ubuntu:

<...>
Apr 17 21:20:04 mc_containers_dev dockerd[27423]: time="2019-04-17T21:20:04.400543977Z" level=debug msg="sandbox set key processing took 674.062994ms for container 85e921cec3464674c1cf141f278e427e12ed1afb29900a240d829549901a06f5"
<...>
Apr 17 21:20:05 mc_containers_dev dockerd[27423]: time="2019-04-17T21:20:05.502389712Z" level=debug msg="sandbox set key processing took 1.701499201s for container 36cac605868411e4091fac50e28ad12b331fbb9d12ec7d7e204f9c248f861f2c"
<...>
Apr 17 21:20:06 mc_containers_dev dockerd[27423]: time="2019-04-17T21:20:06.480814368Z" level=debug msg="sandbox set key processing took 2.744115934s for container a8202cc76af4cda29c1f893d994bdc7a8397621fd389bcfaf9ca0738119290ef"
<...>
Apr 17 21:20:07 mc_containers_dev dockerd[27423]: time="2019-04-17T21:20:07.493168068Z" level=debug msg="sandbox set key processing took 3.604762052s for container ea9a6a394a43adbf9aef2780a53c8ef55a278449c1ec003d752622e7fbe9a9b8"
<...>
Apr 17 21:20:08 mc_containers_dev dockerd[27423]: time="2019-04-17T21:20:08.487253636Z" level=debug msg="sandbox set key processing took 4.608973189s for container e2673ea37fb18b5ba12f9532d47bdbda99a0755ce5219693f9eaf9e45c7dc7ff"
<...>

I’m using Docker 18.09.5 on Ubuntu and 18.09.5 (comes with Docker Desktop 2.0.0.3) on macOS. Docker Compose is at version 1.24.0 on both machines.

So, why docker-compose up's networking is 2x faster on an VM on macOS, and what can I try to make it just as fast when being run natively on Linux?

2

So, why docker-compose up 's networking is 2x faster on an VM on macOS, and what can I try to make it just as fast when being run natively on Linux?

Because your macOS is a Desktop and the Linux machine is a server, therefore hasn’t have enough entropy, and docker-compose is known to hang for a lot of seconds.

The solution:

The Github issue: