I’ve a file server, which allows client to upload files. I created a docker image with the file server and deployed it in my CentOS. It works as expected.
Today, I found a very weird thing:
When I used a VPN at the client side, I can’t send a large file successfully. However, if I deploy the file server (bin file) immediately, I could send any large file successfully.
a) client → VPN → file server with docker container ----- failure on sending large file
b) client → VPN → file server ----- succ
Besides, for the case a), it always spends 20 seconds to get the error info at the client side. I’m using Postman as the client, so if I send a large file, after 20 seconds, Postman would show me:
Could not send request
Error: read ECONNRESET
It seems that there is some limitation on the docker container: if the network is kind of unstable (as we know, VPN is unstable), TCP will be abandon in 20 seconds.
Could you please help me on this issue? Does docker exist such a limitation mechanism?
a) client → VPN → file server with docker container ----- failure on sending large file
b) client → VPN → file server ----- succ
c) client → file server with docker container ----- succ randomly
d) client → file server ----- succ
For the case c), sometimes it could fail, but sometimes it could succeed.
In fact, first of all, after deploying the file server with Docker container, I got the case c) and I don’t know why. And once I enabled my VPN accidently, I got the same phenomenon with 100% failure, which is case a).
So I doubt if " the unstable network + Docker container" caused this issue while using Docker container. Because without Docker container (case b) and case d)), it always succeeds. So it’s kind of just like the Docker container abandon the unstable connection… Since the client always get error back in exact 20 seconds, it’s more likely some config of Docker container…
There is no limit or intentional timeouts on Docker networks. Those are just network bridges. Even if it has an impact on the speed, it shouldn’t fail. Debugging network is always difficult to me, but in many cases the issue was a wrong MTU setting. Docker has a default MTU setting. If the difference between the MTU of your phyisical network and the MTU of the Docker network is big, that can cause random package loss causing low speed or failing connection.
docker run --net host ... works without any error!
I just used tcpdump to capture packets while sending a large file without --net host and I just found “Don’t fragment: Set”, Which means that large packets would be abandoned. That’s why I got such an issue.
So for now, it seems that there are some mechanism: if the network is unstable, set “don’t fragment”, else, don’t set “don’t fragment”. I don’t know it’s the docker mechanism, but executing iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu could solve this issue.
Well, it seems that the command iptables -I FORWARD -p tcp --tcp-flags SYN,RST SYN -j TCPMSS --clamp-mss-to-pmtu measn that don’t send packet bigger than MTU.
In my opinion, the direct reason for this issue is that the client sends some packets, which are bigger than MTU but with DF(don’t fragment), to the docker container.
I’m sure that the client and the file server don’t touch DF, which means that they used the flag(DF) by default. So I still don’t know how I could explain the case a) and the case b), and I don’t know why docker run --net host works but it won’t work without --net host.
Is docker able to set DF through some mechanism???
The host network is not a special network for Docker. It is the lack of network isolation. So if sending packages to the host works, it will work in the container that uses the host network.