Docker Compose Networking Issue: TCP Network Messages Take 30-60 seconds

Open Source Projects Compose
1

Ubuntu 20.04
Docker version 20.10.17, build 100c701
Docker Compose version v2.6.0

I am testing a fully connected mesh network using Docker compose. Beyond a certain number of containers in the network, TCP message sending becomes very laggy. Messages are sent (from the perspective of my code) and then it takes anywhere from 30-60 seconds for those messages to be received by the other container.

I have a reproducing sample of this here. Just execute ./run.sh to start a network that passes messages back and forth demonstrating the problem. Setting an environment variable dictates the number of containers Docker compose creates. Each container runs one copy of the server binary. When this number is below 32, there is no message delay. When the number is 32 or greater, there is significant delay. The issue would be very pronounced when running with 31 nodes vs 75 nodes, for example.

Running exactly the same code with exactly the same number of nodes using a bash script without Docker does not exhibit this problem, up to 780 nodes. All of the messages are received as soon as they’re sent.

The problem is specific to Docker container usage. Does anyone have any ideas on what might be causing this Docker-specific network lag, and how I might debug it?

1 Like
2

I am having a similar issue.
I have a database (not inside the compose file) + an http2 service inside a compose file describing an external network (the same used by the database). When I run “docker compose” to build and start the service, I have 2sec to 3sec latency per request !!
When I run by hand the service (“run” command line), the latency is gone: response time between 250ms & 400ms per request.

I think there is something that is being done in background by docker compose that causes network latency.