Docker Community Forums

Share and learn in the Docker community.

HTTP 408: Request Time-out... but not a ratelimit or DNS issue?

Hello fellow Docker users :wave: I hope you’re well. Been using & enjoying Docker for many years with great success until this week. I’ve got a mystery that I hope you can help me solve.

In the beginning, docker pull stopped working…

It all started with a request timeout, which in practice looks like this:

$ docker pull helloworld
Using default tag: latest
Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"

Everything is a DNS problem, right? Not so fast…

After lots of troubleshooting around whether this might be a DNS issue, I then began to think I might be getting rate limited, so I found and followed these instructions:

The results were… surprising. I can get complete the first step (obtaining an API token):

$ TOKEN=$(curl --user "${DOCKER_USERNAME}:${DOCKER_PASSWORD}"  "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  4307    0  4307    0     0  12098      0 --:--:-- --:--:-- --:--:-- 12098
$ echo $TOKEN
[[ REDACTED ]]

But when I try to complete the next step, to “get the headers showing your limits”, I get another HTTP 408 Request Time-out:

$ curl --head -H "Authorization: Bearer ${TOKEN}" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
HTTP/1.0 408 Request Time-out
cache-control: no-cache
content-type: text/html

It appears it’s NOT actually a DNS problem…

The fact that these requests are to distinct DNS records (auth.docker.io vs registry-1.docker.io) got me thinking again that this might be a DNS issue. However, I discovered (somewhat accidentally) that I can not only resolve DNS, but also get an IMMEDIATE response back from the API:

$ curl --head -H "Authorization: Bearer INVALIDTOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:ratelimitpreview/test:pull",error="invalid_token"
date: Thu, 12 Aug 2021 18:20:18 GMT
content-length: 164
strict-transport-security: max-age=31536000

If I change to a GET (-XGET) request instead of a HEAD (-I or --head) request, I can even get a response body back:

$ curl -XGET -H "Authorization: Bearer INVALIDTOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"ratelimitpreview/test","Action":"pull"}]}]}

Is my IP address on a Docker Hub deny-list? LOL

So apparently I can resolve DNS, and get HTTP 401 Unauthorized responses, but I cannot authenticate and see my ratelimit headers.

So then I thought my IP address might be flagged. My ISP doesn’t offer static IP addresses, so I have a cron job that runs a little script I wrote which updates a personal domain with my dynamic DNS, and it logs to /var/log/ddns.log. According to that log file the last time my ISP changed my IP address was July 10. I’m 100% certain I have pulled at least dozens (if not hundreds) of Docker images since then with no problems. But just in case I’ve somehow ended up on an IP deny list, let me try to get a new IP address. Before calling my ISP I thought I might try rebooting my fiber modem & router, and voila! It worked! No need to talk to the ISP…

$ sudo tail /var/log/ddns.log
Thu 12 Aug 2021 10:55:02 AM PDT nochg xx.xxx.xxx.238
Thu 12 Aug 2021 11:00:41 AM PDT 
Thu 12 Aug 2021 11:05:41 AM PDT 
Thu 12 Aug 2021 11:10:01 AM PDT good xx.xxx.xxx.204

…redacted because reasons :grinning_face_with_smiling_eyes: – I love y’all, but no need to invite prying eyes…

So here I am, posting on hacker news the Docker Community Forums, :crossed_fingers: that someone has any ideas about what the heck is going on.

Thanks in advance for your time!! :v:

Just in case it helps: as you are getting a response from the remote server (being the HTTP 408 error), I’d not look for DNS problems: to be able to connect to that server you clearly got an IP address for the domain name you’re trying to access, so DNS worked, I’d say.

Also, as Docker uses HTTP 429 Too Many Requests when running into rate limits, I’d not first look for rate limit problems when instead seeing a 408 Request Timeout.

Is that the actual command you’re using, with helloworld instead of hello-world? I assume it should not matter for the 408, but what if you try with an image that actually exists?

Aside, searching for 408 on this very forum yields:

I’d assume that for such solution not only Docker would give you problems, but I’ve no idea.

Before diving into that: maybe you could repeat your curl tests with a long value for INVALIDTOKEN (at least the same length as a valid token). And repeat it using the valid token on another computer, preferably on the same network/IP address.

1 Like

Thank you for your reply! A few responses to your comments/questions for posterity:

  • Yes, I agree DNS was not the issue (tried to explain that in my OP)
  • I did see that actual rate limit responses would result return a 429, and I wasn’t seeing a 429, but I was running out of other ideas…
  • I haven’t been having ANY other DNS or timeout issues with any other app/service/etc, which made/makes this even more confounding

Decreasing MTU size to 1000 seems to have resolved my issue. This is incredibly bizarre given that I haven’t made any changes to my network (that I know of); perhaps an auto-update changed something? I’ll have to investigate… Either way, thanks for your help!