Hi, I’ve got a problem when trying to pull images from the official docker registry. Everytime I start a pull it downloads relatively fast at the beginning but rapidly decreases download speeds to about 20-70 kbit/s, so it takes forever to complete (> 30-45 mins depending on which image). This especially happens with large layers.
Everything else is using normal download speeds, so my connection seems to be fine.
I did run mtr which outputs the following after running it for 30 minutes minimum:
I can see that at some point there is a massive loss of packets. Using dig index.docker.io I can see that it should be trying to connect to us-east-1 of aws:
;; QUESTION SECTION:
;index.docker.io. IN A
;; ANSWER SECTION:
index.docker.io. 0 IN CNAME elb-io.us-east-1.aws.dckr.io.
elb-io.us-east-1.aws.dckr.io. 0 IN CNAME us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com.
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 52.55.53.172
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 54.85.207.112
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 34.231.195.77
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 3.231.52.87
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 34.198.0.87
us-east-1-elbio-rm5bon1qaeo4-623296237.us-east-1.elb.amazonaws.com. 0 IN A 34.202.228.80
However using windows resource monitor it seems that vpnkit.exe is accessing e.g. 104.18.123.25 which should be an ip belonging to cloudflare with a download speed of ~20-70 kbit/s, so maybe there is a bottleneck?
Does anyone else have problems like this? Its so slow that I really cannot work with docker at the moment.
Windows 10 21H2 19044.1415
Docker Desktop Version 4.3.2 (72729)
Engine: 20.10.11
Compose: v2.2.1
Can you tell me an example what image you are trying to download so I can try if it is slow for me too? I didn’t notice any speed issue, but it could depend on your location. How long have you been trying to pull images with this speed?
Sorry for the late reply. It has been like this for weeks now, unfortunately I don’t know it more precise. I actually completely reinstalled my system, not necessarily because of this problem, but it didn’t help either.
It happens with all images I tried so far, I just ran pull for postgres:
~ ──────────────────────────────────────────────────────── at 18:58:41
❯ time docker pull postgres
Using default tag: latest
latest: Pulling from library/postgres
a2abf6c4d29d: Pull complete
e1769f49f910: Pull complete
33a59cfee47c: Pull complete
461b2090c345: Pull complete
8ed8ab6290ac: Pull complete
495e42c822a0: Pull complete
18e858c71c58: Pull complete
594792c80d5f: Pull complete
794976979956: Pull complete
eb5e1a73c3ca: Pull complete
6d6360292cba: Pull complete
131e916e1a28: Pull complete
757a73507e2e: Pull complete
Digest: sha256:f329d076a8806c0ce014ce5e554ca70f4ae9407a16bb03baa7fef287ee6371f1
Status: Downloaded newer image for postgres:latest
docker.io/library/postgres:latest
docker pull postgres 0.18s user 0.15s system 0% cpu 5:08.15 total
or jboss/keycloak:
~ ──────────────────────────────────────────────────────── at 19:17:06
❯ time docker pull jboss/keycloak
Using default tag: latest
latest: Pulling from jboss/keycloak
ac10f00499d5: Pull complete
96d53117c12e: Pull complete
6c765d9e2ec8: Pull complete
82c9b130dd92: Pull complete
0a1b7d7fb8e1: Pull complete
Digest: sha256:6ecb9492224c6cfbb55d43f64a5ab634145d8cc1eba14eae8c37e3afde89546e
Status: Downloaded newer image for jboss/keycloak:latest
docker.io/jboss/keycloak:latest
docker pull jboss/keycloak 0.43s user 0.21s system 0% cpu 25:43.00 total
Pulling postgres takes 5:08 minutes and keycloak takes 25:43 minutes
I also think it could have something to do with my location or ISP. I’m in Germany with Telekom as my internet provider. Speedtests are looking good and other applications don’t seem to be affected:
Server: intersaar GmbH - Saarbrucken (id = 3692)
ISP: Deutsche Telekom AG
Latency: 10.33 ms (0.20 ms jitter)
Download: 104.71 Mbps (data used: 85.3 MB )
Upload: 32.03 Mbps (data used: 14.4 MB )
Packet Loss: 0.0%
Both of them are fast for me. From which country are you trying to pull the images? I would try from there (if I can) using Surfshark vpn. If the country is not the issue, then something must be in your network or your internet provider’s network.
I used WSL2 with Ubuntu, I forgot to mention that. But its the same behavior using Powershell.
I’m trying to pull from Germany. I noticed that npm also has that issue, so it can’t be specific to docker. An mtr to registry.npmjs.org showed packetloss with cloudflare, so I ran mtr with cloudflare.com:
I tried from Berlin and Frankfurt. It was about the same speed as without vpn. I also have an ubuntu VM in Frankfurt on DigitalOcean. It is fast too.
Since you have had this issue for weeks and you say it is probably not Docker specific, I would suspect something with your internet provider. Maybe a firewall or bad routing.
I have a similar issue coming from AS3320 DTAG and have extremely slow downloads from DockerHub. My AS path is DTAG (AS3320) <-> SEA-BONE (AS6762) <-> Cloudflare (AS13335) and it seems like the SEA-BONE network might be the issue in my case.
This seems to be a DTAG problem and the choice of peerings they make that influences the rest of the route to the target. I would strongly suggest to raise a support ticket at your isp. There is not much you can do about the peering your isp uses.
As a workaround, a VPN connection with a non DTAG internet breakout might mitigate the problem. I know that it doesn’t solve the issue, but at least it might help to workaround the limitation for time beeing. It’s worth a try - but it also might not help at all.
Hmm, on a second thought it might be more likely a cloudflare problem? Your route uses a different CF hop (in Frankfurt) before the target, then it does for me on VF. Mine uses cloudflare.bcix.de (in Berlin) and speeds are like always.
This could have many reasons. Best bet realy is to complain to you isp.
