Internet access in Docker container is much slower compared to access from machine running docker

Internet access from within a docker container is much slower than without.

I’m running Ubuntu 20.04.6 LTS and docker Docker version 26.1.0, build 9714ad and have the following docker packages installed:

❯ apt list --installed | grep docker
docker-buildx-plugin/focal,now 0.14.0-1~ubuntu.20.04~focal amd64 [installed,automatic]
docker-ce-cli/focal,now 5:26.1.0-1~ubuntu.20.04~focal amd64 [installed]
docker-ce-rootless-extras/focal,now 5:26.1.0-1~ubuntu.20.04~focal amd64 [installed,automatic]
docker-ce/focal,now 5:26.1.0-1~ubuntu.20.04~focal amd64 [installed]
docker-compose-plugin/focal,now 2.26.1-1~ubuntu.20.04~focal amd64 [installed]

Docker info output:

❯ docker info
Client: Docker Engine - Community
 Version:    26.1.0
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.14.0
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.26.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 0
 Server Version: 26.1.0
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: e377cd56a71523140ca6ae87e30244719194a521
 runc version: v1.1.12-0-g51d5e94
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.15.0-105-generic
 Operating System: Ubuntu 20.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 16
 Total Memory: 62.43GiB
 Name: p1
 ID: 33TX:EJKT:EQUI:UYZK:PWFS:S3LI:V3RR:WANC:HIGH:UUZE:7TQP:2ZYP
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

Gathering Evidence

I noticed downloads being particular slow when building and running containers that pull in relatively big artifacts from the internet.

As an experiment I tried running speedtest-cli (apt install speedtest-cli):

  • From my local machine (Ubuntu 20.04.6 LTS):
    ❯ speedtest-cli
    Retrieving speedtest.net configuration...
    Testing from Telenet (84.195.66.19)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by NewsXS B.V. (Amsterdam) [162.03 km]: 24.149 ms
    Testing download speed................................................................................
    Download: 255.20 Mbit/s
    Testing upload speed......................................................................................................
    Upload: 28.52 Mbit/s
    
  • From a docker container (ubuntu:22.04) running on the same local machine
    ❯ docker run -it ubuntu:22.04 /bin/bash
    root@6e877f0efff1:/# apt update && apt install --yes speedtest-cli
    ...
    root@6e877f0efff1:/# speedtest-cli
    Retrieving speedtest.net configuration...
    Testing from Telenet (84.195.66.19)...
    Retrieving speedtest.net server list...
    Selecting best server based on ping...
    Hosted by Claranet Benelux B.V. (Amsterdam) [162.03 km]: 2532.721 ms
    Testing download speed................................................................................
    Download: 2.25 Mbit/s
    Testing upload speed......................................................................................................
    Upload: 2.94 Mbit/s
    

Has anyone run into a similar issue before? And how did you resolve it?

I ran into this potential MTU Issue but I don’t think it’s it, the MTU of my default network interface is the same as the MTU of the docker interface:

❯ ip link | grep mtu
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
2: wlp0s20f3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DORMANT group default qlen 1000
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default 

In my experience, if it’s an MTU issue, larger files (>MTU size) won’t work at all.

I run docker-ce 26.1.0 and have a 1000/200 fiber connection. I use speedtest-tracker - LinuxServer.io to measure the bandwidth. It measures roughly ~940 Mbps down and ~220 Mbps up. The download bandwidth was slightly better before I switched to an OpenWRT based router.

It’s safe to say that it’s not a general docker problem :slight_smile:

1 Like

Thanks for testing, although it doesn’t fully rule out it’s not a docker issue.
There is definitely an issue on my side, and it’s related to docker, so I suspect it might be a configuration problem

I tried a different speedtest tool (Ookla Speedtest) and interestingly now I don’t see any differences anymore:

  • Local machine (Ubuntu 20.04.6 LTS):
   Speedtest by Ookla

      Server: VOO - Liège (id: 23320)
         ISP: Telenet
Idle Latency:     9.20 ms   (jitter: 1.20ms, low: 7.97ms, high: 10.60ms)
    Download:   267.37 Mbps (data used: 281.3 MB)                                                   
                171.00 ms   (jitter: 54.25ms, low: 12.09ms, high: 259.14ms)
      Upload:    28.45 Mbps (data used: 14.6 MB)                                                   
                  5.83 ms   (jitter: 6.24ms, low: 4.83ms, high: 245.06ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/1b9451d0-2f91-4f1d-b591-00a561702329
  • Container on local machine (ubuntu:22.04):
   Speedtest by Ookla

      Server: VOO - Liège (id: 23320)
         ISP: Telenet
Idle Latency:     9.22 ms   (jitter: 2.74ms, low: 8.88ms, high: 14.55ms)
    Download:   267.29 Mbps (data used: 337.5 MB)                                                   
                161.30 ms   (jitter: 50.58ms, low: 15.45ms, high: 259.40ms)
      Upload:    28.42 Mbps (data used: 14.2 MB)                                                   
                  5.94 ms   (jitter: 0.95ms, low: 4.86ms, high: 25.47ms)
 Packet Loss:     0.0%
  Result URL: https://www.speedtest.net/result/c/0652b080-6d49-4ad0-872d-cba0d0c181d5

I went back to speedtest-cli and got the same results (docker container having much slower internet) as before. I run each of them a few times and the results both (speedtest being ok and speedtest-cli being slow in the container but not on my local machine seem to be consistently replicable)

I’ve also been able to replicate the issue by running traceroute:

  • Local machine (Ubuntu 20.04.6 LTS):
    • Traceroute runs for 6 seconds
      ❯ command time --verbose traceroute google.com
      traceroute to google.com (64.233.166.100), 30 hops max, 60 byte packets
       1  _gateway (192.168.0.1)  1.655 ms  3.760 ms  4.056 ms
       2  * * *
       3  * * *
       4  * * *
       5  72.14.204.36 (72.14.204.36)  28.888 ms  27.284 ms  30.665 ms
       6  * * *
       7  142.251.246.110 (142.251.246.110)  23.940 ms 209.85.254.146 (209.85.254.146)  26.236 ms 209.85.255.16 (209.85.255.16)  27.916 ms
       8  192.178.86.8 (192.178.86.8)  28.258 ms 142.251.224.8 (142.251.224.8)  17.760 ms 142.251.64.32 (142.251.64.32)  20.295 ms
       9  64.233.174.221 (64.233.174.221)  21.548 ms 172.253.65.169 (172.253.65.169)  26.372 ms 192.178.85.69 (192.178.85.69)  17.853 ms
      10  209.85.142.243 (209.85.142.243)  16.585 ms 142.251.67.24 (142.251.67.24)  18.357 ms 72.14.234.184 (72.14.234.184)  18.651 ms
      11  216.239.57.143 (216.239.57.143)  21.092 ms * 216.239.51.165 (216.239.51.165)  52.215 ms
      12  * * *
      13  * * *
      14  * * *
      15  * * *
      16  * * *
      17  * * *
      18  * * *
      19  * * *
      20  * * *
      21  wm-in-f100.1e100.net (64.233.166.100)  72.453 ms  70.728 ms  74.500 ms
      	Command being timed: "traceroute google.com"
      	User time (seconds): 0.00
      	System time (seconds): 0.00
      	Percent of CPU this job got: 0%
      	Elapsed (wall clock) time (h:mm:ss or m:ss): 0:06.75
      	Average shared text size (kbytes): 0
      	Average unshared data size (kbytes): 0
      	Average stack size (kbytes): 0
      	Average total size (kbytes): 0
      	Maximum resident set size (kbytes): 3352
      	Average resident set size (kbytes): 0
      	Major (requiring I/O) page faults: 0
      	Minor (reclaiming a frame) page faults: 186
      	Voluntary context switches: 117
      	Involuntary context switches: 0
      	Swaps: 0
      	File system inputs: 0
      	File system outputs: 0
      	Socket messages sent: 0
      	Socket messages received: 0
      	Signals delivered: 0
      	Page size (bytes): 4096
      	Exit status: 0
      
  • Container on local machine (ubuntu:22.04):
    • traceroute takes 1.5 mintues
      root@2a7980678092:/# command time --verbose traceroute google.com
      traceroute to google.com (74.125.133.101), 30 hops max, 60 byte packets
       1  172.17.0.1 (172.17.0.1)  0.018 ms  0.006 ms  0.004 ms
       2  192.168.0.1 (192.168.0.1)  2.122 ms  2.433 ms  3.283 ms
       3  * * *
       4  * * *
       5  * * *
       6  72.14.204.36 (72.14.204.36)  24.574 ms  16.166 ms  21.532 ms
       7  * * *
       8  209.85.254.116 (209.85.254.116)  21.616 ms 209.85.248.158 (209.85.248.158)  21.604 ms 142.251.246.110 (142.251.246.110)  22.981 ms
       9  142.251.224.10 (142.251.224.10)  19.462 ms 142.251.64.32 (142.251.64.32)  28.502 ms  26.675 ms
      10  108.170.236.237 (108.170.236.237)  22.901 ms 64.233.174.221 (64.233.174.221)  24.671 ms 142.251.247.79 (142.251.247.79)  27.550 ms
      11  142.251.61.121 (142.251.61.121)  31.195 ms 142.251.61.123 (142.251.61.123)  27.336 ms 142.251.71.167 (142.251.71.167)  81.554 ms
      12  209.85.243.33 (209.85.243.33)  26.492 ms 209.85.243.35 (209.85.243.35)  26.453 ms 209.85.241.237 (209.85.241.237)  21.169 ms
      13  * * *
      14  * * *
      15  * * *
      16  * * *
      17  * * *
      18  * * *
      19  * * *
      20  * * *
      21  * * *
      22  wo-in-f101.1e100.net (74.125.133.101)  26.409 ms  28.159 ms  28.152 ms
      	Command being timed: "traceroute google.com"
      	User time (seconds): 0.00
      	System time (seconds): 0.01
      	Percent of CPU this job got: 0%
      	Elapsed (wall clock) time (h:mm:ss or m:ss): 1:35.97
      	Average shared text size (kbytes): 0
      	Average unshared data size (kbytes): 0
      	Average stack size (kbytes): 0
      	Average total size (kbytes): 0
      	Maximum resident set size (kbytes): 2912
      	Average resident set size (kbytes): 0
      	Major (requiring I/O) page faults: 0
      	Minor (reclaiming a frame) page faults: 128
      	Voluntary context switches: 74
      	Involuntary context switches: 0
      	Swaps: 0
      	File system inputs: 0
      	File system outputs: 0
      	Socket messages sent: 0
      	Socket messages received: 0
      	Signals delivered: 0
      	Page size (bytes): 4096
      	Exit status: 0
      

I tried running with privileged (as suggested here) : docker run -it --privileged ubuntu:22.04 /bin/bash, but also no success.

Ok, I think it’s a docker DNS issue, adding --dns "8.8.8.8" seems to fix the issue:

❯ docker run -it --dns "8.8.8.8" ubuntu:22.04 /bin/bash
...
root@bb95c7e0a737:/# speedtest-cli
Retrieving speedtest.net configuration...
Testing from Telenet (84.195.66.19)...
Retrieving speedtest.net server list...
Selecting best server based on ping...
Hosted by Claranet Benelux B.V. (Amsterdam) [162.03 km]: 29.612 ms
Testing download speed................................................................................
Download: 210.06 Mbit/s
Testing upload speed......................................................................................................
Upload: 28.65 Mbit/s

Using the host network driver --net host also results in the same issue. Only explicitly setting the DNS seems to work.

Now I still have to figure out how to apply this --dns option to docker build

Not sure why Docker has this DNS issue and slowdown. If anyone can help me understand, than this would be appreciated :pray:.

1 Like

Based on this post the setting the /etc/docker/daemon.json to:

{
    "dns": ["8.8.8.8"]
}

followed by sudo service docker restart seems to work.

Please note that I cannot have 10.0.0.2 listed in the dns list, or I will experience the issues mentioned

2 Likes

This is caused by the dns setting of your host. While containers attached to the default bridge network will mount the host’s /etc/resolv.conf, containers attached to a user defined bridge network will use docker’s internal dns server, which uses the configuration from the host’s /etc/resolv.conf as upstream.

The setting you made just bypasses the default behavior, which shouldn’t be necessary if the host’s configuration is okay.

Personally, I would investigate what dns resolver was configured and fix the problem at the source. For instance, on Ubuntu I use systemd-resolved without the stub resolver, so that my /etc/resolv.conf actually points to a dns resolver available in my network. The stub resolver can be deactivated in the responsible config file in /etc/systemd/resolved.conf.d/ and then requires a restart of the service systemctl restart systemd-resolved.service.

4 Likes

It seems like the MTU (Maximum Transmission Unit) of both your default network interface and the Docker interface is set to 1500, which suggests that an MTU issue might not be the cause of your problem. The MTU defines the maximum size of a data packet that can be transmitted over a network interface without fragmentation.

In your case, the MTU values for both interfaces are the same:

  1. The default network interface (wlp0s20f3) has an MTU of 1500.
  2. The Docker interface (docker0) also has an MTU of 1500.

Since the MTU values match and are within the typical range for Ethernet networks, it’s less likely that an MTU-related issue is causing your problem.

If you’re experiencing network-related issues, it might be worth exploring other potential causes or troubleshooting steps to diagnose and resolve the issue.

Let me know if you need further assistance or if there’s anything else I can help you with!

1 Like