Rate limit reached (compose plugin), but I'm not pulling images at all

Docker Hub
,
1

For a week or so, whenever I pull an image from Docker Hub via a docker compose file, I am told that I’ve reached my 'pull limit '.

If I authenticate (to get 200 pulls), I get the same message, that I reached the pull limit.

I also tried switching IP addresses (new server). But no luck, same message.

However, if I do a direct pull (just with the docker command), I can pull any image, without any issues.

Does anyone have an idea what’s going on here? This issue just appeared out of the blue for me, and by now I’ve tried everything I could find, to get to the bottom of it.

Besides that, I’m never pulling more than 5 images every few weeks …

I’m using:

  • Docker Compose version v2.29.7
  • Docker version 27.3.1, build ce12230
  • Ubuntu 22.04 on a DO VPS
2

So this issue seems ot be complex.

  1. Your are not pulling images, yet you get the pull limit error
  2. You get the same error even without logging in and also after logging in
  3. You get the same error message on another server
  4. You get the error only from Docker Compose which shouldn’t matter

Hours ago there was an issue with Docker Hub, so it is either that Docker Hub is still not fully funtional and Docker Compose does something that isn’t used by the docker command, or you use the same Docker Compose version on all machines and that Compose version has a bug. Have you tried updating Docker Compose?

3

This has been going on for a week and a half now. I’ve also read that docker compose pulls in ‘different’ ways from the docker command, but that doesn’t explain a sudden change of behavior, within a context that doesn’t justify a rate limit hit (I’m not even pulling more than a few images a month).

I’m using the latest docker versions and the compose plugin, so a downgrade could be worth a try. Then again, it worked before, so it must be something else …

4

The latest Docker Compose is v2.30.3. This is what I have on macOS installed by Docker Desktop, but I see that yours is the latest in the APT repository. So you can downgrade or upgrade if you download the release directly from GitHub

I’m not sure why the latest is not added to the APT repository yet

5

I have a similar bug: for a week, I can’t pull any image from our staging environment via docker-compose but after reading this post, I could do a simple docker pull.

docker pull postgres:14.13
14.13: Pulling from library/postgres
2d429b9e73a6: Already exists 
e19d151f086a: Pull complete 
8bcada4d21a4: Pull complete 
f208fc972da5: Pull complete 
3c98c3576ddb: Pull complete 
f588171285a1: Pull complete 
40693100d656: Pull complete 
df8d3deaa47c: Pull complete 
d86466467f89: Pull complete 
6ee9a2e499ca: Pull complete 
5cb7031207a4: Pull complete 
8d484f1e7989: Pull complete 
cb05aa8d998c: Pull complete 
4827700d7c77: Pull complete 
Digest: sha256:9fcd27ebadf8d8fb1f257b1799eff6363cb0d947ffff046de5d49915f469ca08
Status: Downloaded newer image for postgres:14.13
docker.io/library/postgres:14.13

but:

docker compose pull

[+] Pulling 15/15
 ✔ mqttlistener Skipped - Image is already being pulled by worker_mqtt_telemetry                                     0.0s 
 ✔ worker_stream Skipped - Image is already being pulled by worker_mqtt_telemetry                                    0.0s 
 ✔ worker_abnormal_consumption Skipped - Image is already being pulled by worker_mqtt_telemetry                      0.0s 
 ✔ worker_command Skipped - Image is already being pulled by worker_mqtt_telemetry                                   0.0s 
 ✔ worker Skipped - Image is already being pulled by worker_mqtt_telemetry                                           0.0s 
 ✔ api Skipped - Image is already being pulled by worker_mqtt_telemetry                                              0.0s 
 ✔ worker_aggregation Skipped - Image is already being pulled by worker_mqtt_telemetry                               0.0s 
 ✔ worker_mqtt Skipped - Image is already being pulled by worker_mqtt_telemetry                                      0.0s 
 ✔ postgres Pulled                                                                                                   0.8s 
 ✔ worker_mqtt_telemetry Pulled                                                                                      0.6s 
 ✔ warp10 Pulled                                                                                                     0.5s 
 ✔ mqtt Pulled                                                                                                       0.9s 
 ✘ redis Error                                                                                  toom...              1.1s 
 ✔ webapp Pulled                                                                                                     0.6s 
 ✔ backup Pulled                                                                                                     0.5s 
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

Above, redis, postgres and emqx are hosted on docker hub. redis and postgres are official images whereas emqx which is pulled is from a user repo emqx/emqx and not the official image. All other images are pulled from our internal gitlab registry.

We are not authenticated on docker hub as we don’t have any subscriptions yet.

docker compose pull mqtt

[+] Pulling 1/1
 ✔ mqtt Pulled

but:

docker compose pull redis

[+] Pulling 1/1
 ✘ redis Error toomanyrequests: You have reached your pull rate limit. You may increase the limit by...              1.1s 
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

As I just pulled postgres via docker pull it now works again:

docker compose pull postgres

[+] Pulling 1/1
 ✔ postgres Pulled

but before that it failed.

dpkg -l docker*
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                      Version                       Architecture Description
+++-=========================-=============================-============-================================================>
un  docker                    <none>                        <none>       (no description available)
ii  docker-buildx-plugin      0.17.1-1~debian.11~bullseye   amd64        Docker Buildx cli plugin.
ii  docker-ce                 5:27.3.1-1~debian.11~bullseye amd64        Docker: the open-source application container en>
ii  docker-ce-cli             5:27.3.1-1~debian.11~bullseye amd64        Docker CLI: the open-source application containe>
ii  docker-ce-rootless-extras 5:27.3.1-1~debian.11~bullseye amd64        Rootless support for Docker.
ii  docker-compose-plugin     2.29.7-1~debian.11~bullseye   amd64        Docker Compose (V2) plugin for the Docker CLI.
un  docker-engine             <none>                        <none>       (no description available)
un  docker.io                 <none>                        <none>       (no description available)
lines

And containerd.io 1.7.23-1

From our CI server we have no troubles at all whereas it pulls more image than our staging environment.

So are there some special restrictions on official images that are less or not present on user based images ?

6

For apt repo not up to date, it’s because it’s tied with a new docker release - cf [BUG] sudo apt-get install docker-compose-plugin OR sudo apt update does not pull latest compose version. Want 2.30.2 Installed via APT · Issue #12269 · docker/compose · GitHub

1 Like
7

This is probably because of the “overall rate limit”

Other limits

Docker Hub also has an overall rate limit to protect the application and infrastructure. This limit applies to all requests to Hub properties including web pages, APIs, and image pulls. The limit is applied per-IP, and while the limit changes over time depending on load and other factors, it’s in the order of thousands of requests per minute. The overall rate limit applies to all users equally regardless of account level.

You can differentiate between these limits by looking at the error code. The “overall limit” returns a simple 429 Too Many Requests response. The pull limit returns a longer error message that includes a link to this page.

8

Yet, I’ve changed servers (different IPs) and got the same error back. I’ve also tried a different docker engine version, but it gives me the same issue.

It’s a mystery, for now, so I’m continuing to pull directly, which does work …

9

Did you have “toomanyrequests” in your error message as well? I assumed you had a different one. If yours is different, can you share the full error message?

You can also report the issue in the compose github repository

https://github.com/docker/compose/issues

10

But what I don’t get:

  • Our staging environment do between 0 and 3 deployments a day and it faces this issue
  • Yesterday, the only pull that were made were the one I did manually - So for 3 images coming from the hub with let’s say 10 layers in average, it would be below 50 calls per day (with few overhead to get digests and so on)
  • For the last 2 years, it worked well and we deployed more often per day in the past than we do now
  • Our CI servers do more pull than our staging server and does not face any issue on rate limiting whereas it should be ther first one to hit the limit.

So for me there is something wrong somewhere since 2 weeks.

I thought first that docker hub would ban some IP ranges instead of unique IPs and that I was caught in the middle of “noisy neighboors”.

Is there a way to know how much requests a given IP made against docker hub ? I would be interested to dive deeper and understand what leads to this issue.

In the meantime, we’ll push official images from docker hub to our internal registry so that deployments can be made smoothly and be only dependant from our gitlab registry.

11

Does that environment has a unique IP address for outgoiing traffic? If there is a single IP for multiple servers, Docker Hub will see that single IP and your server will be blocked even if another server pulled a lot of images.

Could be. Since you also wrote docker pull worked (I missed that), I can suggest the same github repository to report the issue as I suggested before.

I don’t think so. Docker staff members could check it probably. If you report the issue on GitHub, they can tell you what the next step is. Please, if any of you open an issue, share the link here so everyone else can find it, including us. In the last issue I saw, it turned out there was a problem with Docker Hub, although it was not about requests, but a staff member could check the access logs and start to investigate.

12

Yes … I get the exact same error all across my tests, regardless of IP, server, etc.

13

Then I was wrong assuming you have a different problem. Please, always share the exact error messages :slight_smile: I hope it wil be resolved soon.

14

Does that environment has a unique IP address for outgoiing traffic? If there is a single IP for multiple servers, Docker Hub will see that single IP and your server will be blocked even if another server pulled a lot of images.

No, we have only dedicated IP per server - no IP sharing.

I’ll open an issue on docker/compose repo tomorrow - I wasn’t sure so far it was a docker compose only issue and not a docker hub one.

Let’s see what happens.

Just gave a try with no pull for today;

docker compose pull redis

[+] Pulling 1/1
 ✘ redis Error toomanyrequests: You have reached your pull rate limit. You may increase the limit by...              1.1s 
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit

docker pull redis:7.2.6
Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit
15

Issue created : [BUG] Rate limit hit with docker compose · Issue #12336 · docker/compose · GitHub

1 Like
16

So it’s not a docker/compose issue - now we need to know if it’s a docker/cli issue or a docker-hub only issue.

Useful link from my issue in docker/compose: Usage and rate limits | Docker Docs

Once authenticated, you can get some stats about your usage of the docker hub. It may help.

Unfortuneately, I just moved all official images to our private gitlab registry - so on the short term, I’m not to dive deeper on this topic.

1 Like
17

So, now there are some new weird things happening.

On another VPS (a different IP completely) I cannot docker pull mariadb (or any other version) with a direct pull.

It gives me :

Error response from daemon: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: Understanding Your Docker Hub Rate Limit | Docker

Then I go and do a direct pull for Shlink (link shorter) with docker pull shlinkio/shlink:stable.

And it pulls the image without issue …

Same mariadb image pulls without a problem when I direct pull, from yet another VPS …

18

I tried to pull mariadb from 2 different IPs. 2 physical nd 1 VM. All had different Docker version. I tried with logging in and without it. I know it is not a really good test, but I hoped I could get the same error in at least one test.

At the beginning it seemed to me that it was a compose issue, but the other issue I wrote about before was about counters on Docker Hub related to number of pulls. So maybe another counter had a similar issue which causes showing toomanyrequests.

So if it is a Hub issue, there is another repository where it could be reported

I’m really curious what it could be. I pulled official Docker images recently on multiple machines and I had no problem. Maybe I was just lucky.

19

This topic was automatically closed 10 days after the last reply. New replies are no longer allowed.