The scp download (but not upload) speed is very slow from a container to a external pc over the mesh network, is there a way to configure the mesh network to be faster?
Background:
We have a docker swarm network with multiple nodes where we expose a unique port for each user of our system so they can ssh into their own container.
However, when downloading files with scp or rsync (over ssh) the download speed are very low (100 KB/s from a 10 GBps link).
Uploading over scp/rsync works fine (max out the network connection) so it is only in downlink it is a issue.
I have ruled out:
Disk issues - the file in the container is hosted on a fast SSD m2 disk
CPU/MEM issues - no high cpu/mem usage on the node hosting the container
The setup is from a docker swarm perspective pretty standard but we have increased the ingress network from /24 to /16 to overcome the max number of published ports problem (as in our case it restricted the number of concurrent users to much). Currently we have 27 active users.
I cannot claim to be a network expert, so I would expect the speed to be the same in both direction which is clearly not the case.
Have you also checked the disk on the client side to which you download the files? I don’t think that is the problem, but I have to ask to really rule out that. I also assume you tried to download files to multiple machines and the download speed was always slow, right?
Can you try to run a simple Docker container without Swarm? Or is it what magic-wormhole did? I would test the scp with different Docker networks like the default Docker bridge, a custom docker bridge like the one that docker compsoe creates, amnd maybe also with host network if that is possible in your environment for testing purposes.
You can also check your networks on the host and make sure there is no big differentce between the MTU settings.
And my last idea for now is using tcpdump or tshark to see if there is any rejected or retried package in the network traffic that can slow it down.
Yes also the disk on the client side is “fast” (tested on different clients as well) and since I tested the workaround (wormhole) on the same client and got good speed I am pretty sure it is not a client limitation.
Wormhole is a software that (somewhat simplified) creates a “p2p tunnel” (much like webrtc) between two computers to eg send files. wormhole is not aware of that it runs in docker or use any docker specific “tricks”, I only used it to see if I could avoid the download limitation by scp.
We will do some test in the coming days and see if what tcpdump can show. However, unless there is some other way to share a port in docker swarm network (with no direct access to the swarm workers for clients) than to route it over the ingress network, we need to use a mesh network.
I wouldn’t recommend alternative networks just because one is slower than expected, so the network tests are only for figuring out where the real issue is.Then we can point you to the right direction to report the issue if we can’t help.