Slow responses from a web app in swarm cluster

Docker Engine Swarm
1

Hello,
I have some kind of a problem with my docker set-up.
Here is the case:
I have a web-app (java spring-boot). The app servers JSON
I also have angular app which calls this API.

The problem i see, is that the response times are almost double when the app is running inside the swarm cluster.

If the app is started in stand-alone docker container, it works fine(almost the same as when started on the command line w/o docker at all)

BUT
When the app is running as a service in swarm cluster, my response times are almost double compared to the stand-alone app and stand-alone docker container.
The application is 100% the same in stand-alone and in swarm mode(one and the same JAR file)

The response sizes vary from ~3MB to few KB. The responses are slow even with very small response sizes…

From what i see/understand, the difference is the network type. In stand-alone docker container it is bridge, and in swarm mode it is overlay.

For not this is a show stopper for me as the responses in swarm mode exceed ~750ms compared to ~400ms in stand-alone mode.

Has anyone had this problem ?
Any ideas what can i do to solve it ?

2

ı definetly having same issue one angular app and one aspnet 4.5 web api . my resposes are too slow :frowning: have you fixed this issue or found a workaround for this ?

3

When you publish a port in swarm, not only the additional overhead for the overlay network needs to be considered, but also the overhead for the ingress routing mesh.

If you don’t need the routing mesh, you can try to bypass it:

ports:
  - target: 80
    published: 8080
    protocol: tcp
    mode: host

see: https://docs.docker.com/compose/compose-file/#long-syntax-1

If you don’t rely on multi-container on multi-host communication, you can even skip the overlay network:

version: "3.7"
services:
  web:
    networks:
      hostnet: {}

networks:
  hostnet:
    external: true
    name: host

see: https://docs.docker.com/compose/compose-file/#host-or-none

4

I kind of doubt that overlay-network overhead with responses from 5kb to 5mb will double response times linerarly, and i also doubt that the solution is to think about wether you actually need the mesh, or rather expose the app directly on port 80.

I actually do not understand why someone would even give such a response to this question.

Anyway. I am experiencing also performance problems and I am these days investigating on it. it can be many factors. I will than post my findings for my case as answer to this topic and hope that this is maybe helpful once for somebody that doesnt want to bypass swarm entirely. Specially since this thread is ranking high on google.

5

Please extend why this suggestion is bad and would not be a solution that actually reduces latency. I am happy to learn something new!

And please keep in mind the difference hardware/infrastructure had back then :slight_smile:

6

Hey meyay,

the thing that comes short on your answer is, that it can be many factors that cause the latency on swarm and it does not sound to me that network overhead of swarm can cause a doubling of a 3mb json response. It might be that there is some db involved and the db-access is not well configured and in such case bypassing swarms internal addresses would be a very drastic intervention.

Or imagine the swarm is build up on 5 nodes, which are far away from each other, or communicate via internet and not local network. I mean this can cause be the underlying problem as well.

In the given case, it almost sounds like we are talking about a single node testing where I doubt that an additional network hop or some ingress-load-balancing will justifies doubled latency on a 3MB download. or do you think that could be actually the case?

In any case, I had experienced similar sudden performance falls on my rollout on swarm just yesterday and what i meant is that I will post my findings in this post, as it ranks very high on google and I was disappointed to not find more performance tweaks than bypassing ingress and internal container load-balanicing, … on the first link. Doing so is quite a hammer to me. this comes with significant trade-offs depending on the application served and its architecture. But let’s see.

7

Thank you for sharing your thoughts.

I wish there was a response back then. Topics live from the interaction (that never happened in this case, until you responded). I am glad that you shared your thoughts, I wish others that stumbled across the topic would have done the same.

The circumstances that lead to longer response time will remain an unresolved mystery. We don’t know whether this service had to interact with other components, or if the cluster was running in a region of a hyperscaler, or across regions of a hyperscaler or in a homelab using old equipment. Since it was never shared we simply don’t know.

Of course there could be various reasons for the longer response time. Misaligned mtu sizes amongst the nodes, or high latency network connections (which raft was never designed for) could have been a reason. Back then an increase of 350ms appeared like something that could easily be caused by the additional hoops of the ingress routing mesh, the service vip, and if necessary cross-node routing.

I still have one swarm cluster where i still deploy the traefik instances as global service and publish host ports. I am not sure if it still provides an advantage. Back then it reduced response time and helped to retain the original information of the client connection. I still use it like this, even though the hardware and network bandwidth are way beefier than back then.

8

As promised, I wanted to share my findings.

First of all, I have to agree with @meyay. Using host mode removes the routing mesh overhead and is generally the right choice for production. From my testing, though, once that overhead is removed, the dominant source of latency appears to be the network latency itself (cross-node or cross-region hops), rather than Docker Swarm.

For reference, this is the application I tested:

  • Nginx reverse proxy (domain-based routing)
  • FrankenPHP + Symfony application
  • PostgreSQL cluster (Patroni, 3 nodes: 1 primary + 2 replicas)
  • Each application replica only queries its local database replica (no cross-node DB reads)

Almost every request performs at least one database query, since our full-page cache is stored in PostgreSQL.

1. Swarm vs. docker-compose

Running the exact same stack in Swarm (host mode) and with plain docker-compose showed no measurable difference.

We ran extensive load tests (500 requests/sec for 2 minutes from another AWS instance in the same Network) as our project required it. In both cases the average response time overall was around 5 ms.

2. Routing mesh overhead

I also tested a 3-node Swarm cluster using Multipass VMs on my local machine.

Host mode (no routing mesh):

  • Min: 3 ms
  • Max: 9 ms
  • Mean: 4.8 ms

Routing mesh enabled (Nginx + app both accessed through Swarm services):

  • Min: 3 ms
  • Max: 38 ms
  • Mean: 5.8 ms

The maximum latency varied quite a bit between runs, but the average difference stayed around 1 ms. I certainly couldn’t reproduce anything close to a 2× slowdown from todays perspective.

Of course, this was on a local machine, so there was almost no real network latency.

3. Why host mode still matters

The biggest benefit of host mode is network locality.

If your cluster spans multiple regions (for example Berlin, Helsinki and Lisbon), you don’t want a request entering Berlin only to be forwarded by the routing mesh to an application container in Helsinki. That extra network hop will dominate any routing mesh overhead.

With host mode, traffic can stay local (e.g. Nginx → App → DB on the same node). The trade-off is that you have to handle locality and failover yourself.

In our case, we added a health check for the entire node (Nginx + App + DB). If a node is unhealthy, the external load balancer routes traffic to another healthy node. Our dev-op tought us this.

For database writes, however, we intentionally route traffic to the node hosting the PostgreSQL primary. In that case, a little extra latency is acceptable.

Summary

From my testing, I couldn’t reproduce the 2× latency mentioned in this thread. If all nodes are in the same datacenter or AWS Availability Zone, the routing mesh overhead seems quite small (maybe around 1 ms in my tests).

If your nodes are geographically distributed, I would definitely recommend host mode—not because of the routing mesh itself, but because it preserves locality and avoids unnecessary cross-region network hops. In my opinion, those network hops are likely to have a much bigger impact than Swarm’s own overhead.

@meyay If you ever have some spare time to confirm my results, I’d be happy. I fortunately had the time due to the current project I am working on.

Kind regards
Rozbeh Chiryai Sharahi (Roberto)

9

Usually you would not run a Docker Swarm on nodes spanning a large geographic area, as latency could also disturb the raft algo, that handles Swarm leader election.