Hello fellow Docker users 
I hope you’re well. Been using & enjoying Docker for many years with great success until this week. I’ve got a mystery that I hope you can help me solve.
In the beginning, docker pull stopped working…
It all started with a request timeout, which in practice looks like this:
$ docker pull helloworld
Using default tag: latest
Error response from daemon: error parsing HTTP 408 response body: invalid character '<' looking for beginning of value: "<html><body><h1>408 Request Time-out</h1>\nYour browser didn't send a complete request in time.\n</body></html>\n\n"
Everything is a DNS problem, right? Not so fast…
After lots of troubleshooting around whether this might be a DNS issue, I then began to think I might be getting rate limited, so I found and followed these instructions:
The results were… surprising. I can get complete the first step (obtaining an API token):
$ TOKEN=$(curl --user "${DOCKER_USERNAME}:${DOCKER_PASSWORD}" "https://auth.docker.io/token?service=registry.docker.io&scope=repository:ratelimitpreview/test:pull" | jq -r .token)
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 4307 0 4307 0 0 12098 0 --:--:-- --:--:-- --:--:-- 12098
$ echo $TOKEN
[[ REDACTED ]]
But when I try to complete the next step, to “get the headers showing your limits”, I get another HTTP 408 Request Time-out:
$ curl --head -H "Authorization: Bearer ${TOKEN}" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
HTTP/1.0 408 Request Time-out
cache-control: no-cache
content-type: text/html
It appears it’s NOT actually a DNS problem…
The fact that these requests are to distinct DNS records (auth.docker.io vs registry-1.docker.io) got me thinking again that this might be a DNS issue. However, I discovered (somewhat accidentally) that I can not only resolve DNS, but also get an IMMEDIATE response back from the API:
$ curl --head -H "Authorization: Bearer INVALIDTOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
HTTP/1.1 401 Unauthorized
content-type: application/json
docker-distribution-api-version: registry/2.0
www-authenticate: Bearer realm="https://auth.docker.io/token",service="registry.docker.io",scope="repository:ratelimitpreview/test:pull",error="invalid_token"
date: Thu, 12 Aug 2021 18:20:18 GMT
content-length: 164
strict-transport-security: max-age=31536000
If I change to a GET (-XGET) request instead of a HEAD (-I or --head) request, I can even get a response body back:
$ curl -XGET -H "Authorization: Bearer INVALIDTOKEN" "https://registry-1.docker.io/v2/ratelimitpreview/test/manifests/latest"
{"errors":[{"code":"UNAUTHORIZED","message":"authentication required","detail":[{"Type":"repository","Class":"","Name":"ratelimitpreview/test","Action":"pull"}]}]}
Is my IP address on a Docker Hub deny-list? LOL
So apparently I can resolve DNS, and get HTTP 401 Unauthorized responses, but I cannot authenticate and see my ratelimit headers.
So then I thought my IP address might be flagged. My ISP doesn’t offer static IP addresses, so I have a cron job that runs a little script I wrote which updates a personal domain with my dynamic DNS, and it logs to /var/log/ddns.log. According to that log file the last time my ISP changed my IP address was July 10. I’m 100% certain I have pulled at least dozens (if not hundreds) of Docker images since then with no problems. But just in case I’ve somehow ended up on an IP deny list, let me try to get a new IP address. Before calling my ISP I thought I might try rebooting my fiber modem & router, and voila! It worked! No need to talk to the ISP…
$ sudo tail /var/log/ddns.log
Thu 12 Aug 2021 10:55:02 AM PDT nochg xx.xxx.xxx.238
Thu 12 Aug 2021 11:00:41 AM PDT
Thu 12 Aug 2021 11:05:41 AM PDT
Thu 12 Aug 2021 11:10:01 AM PDT good xx.xxx.xxx.204
…redacted because reasons 
– I love y’all, but no need to invite prying eyes…
So here I am, posting on hacker news the Docker Community Forums, 
that someone has any ideas about what the heck is going on.
Thanks in advance for your time!! 