Docker Engine ignoring my daemon.json registry-mirrors entry

[aguedeney]

I am using a Nexus 3 docker proxy configuration. I want to use the Nexus 3 docker proxy as a mirror. The default is set to be a mirror in the /etc/docker/daemon.json as follows:
I obviously using http (insecure) and not https. I systemctl reload docker && systemctl restart docker.

{
  "experimental": true,
  "insecure-registries": [],
  "registry-mirrors": [
    "http://localhost:8082"
  ]
}

I’m able to login using my username and password and qualifying the local repo as follows:

docker pull localhost:8082/alpine:latest

The docker pull is executed on the same server. This works just fine without a problem.

However, if I pull the image without qualifying the repo location it goes directly to dockerhub for alpine:latest all the time which is not desirable. I browsed the proxy cache on nexus 3 and did not see it cached.

My docker server is on Ubuntu Ubuntu 22.04.4 LTS with all the latest updates

Client: Docker Engine - Community
 Version:           26.1.2
 API version:       1.45
 Go version:        go1.21.10
 Git commit:        211e74b
 Built:             Wed May  8 13:59:59 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          26.1.2
  API version:      1.45 (minimum version 1.24)
  Go version:       go1.21.10
  Git commit:       ef1912d
  Built:            Wed May  8 13:59:59 2024
  OS/Arch:          linux/amd64
  Experimental:     true
 containerd:
  Version:          1.6.31
  GitCommit:        e377cd56a71523140ca6ae87e30244719194a521
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

[rimelek (Ákos Takács)]

Your title mentines “insecure repo entry”, but your issue is related to “registry-mirrors”. Using HTTP is indeed considered insecure, but I think you should get an error even when you log in. I don’t remember.

Have you checked the docker daemon logs?

PS: I edited your post to use code blocks for code instead of quotes. You can find some tips here how you can use code blocks so your code will be formatted properly.

How to format your forum posts

The Docker forum is using Discourse, which supports several ways to format your posts: Markdown; see commonmark.org/help for an interactive tutorial BBCode A small selection of HTML Any combination of the above Blank lines for formatting and readability Quite often you’ll need an empty line above specific formatting to make it work. You also need to type blank lines to create proper paragraphs; just pressing Return once and then typing a new block of text right below a previous block only …

[aguedeney]

Thank you for the markup suggestions and corrections. I’ll have a look at the logs today. Where would I need to look to see the docker daemon log file(s)?

This is what my ~/.docker/config.json looks like:

{
        "auths": {
                "localhost:8082": {
                        "auth": "..."
                }
        },
        "aliases": {
                "builder": "buildx"
        }
}

BTW, this was successful:

docker login http://localhost:8082 -u <username> -p <password>

But, unfortunately, the docker pull completely ignored it and failed to pull from Nexus 3’s Docker proxy and went directly to Dockerhub.

To view docker logs on my version of Ubuntu (see above): journalctl -fu docker.service

I now have a better idea of what’s going on. I’m writing some notes to help others that want to use an insecure (http) connection on their localhost for building docker containers (probably a common use case).

In my case, this my example problem:

1. This forces docker to read the /etc/docker/daemon.json: sudo systemctl reload docker
2. The journalctl -fu docker.service allow us to, in fact, see docker reading /etc/docker/daemon.json
3. I grab the error’s url from this error:

dockerd[393960]: time="2024-05-16T21:04:22.885264587-05:00" level=info msg="Attempting next endpoint for pull after error: Head \"http://localhost:8082/v2/library/mysql/manifests/latest\": unauthorized: access to the requested resource is not authorized" spanID=ab8022fec9d8c4d6 traceID=d74b8485ca0441fbffe4e95619b1fd72
4. Now, know that the issue is an authorization problem, I did this:
curl 'http://<username>:<password>@localhost:8082/v2/library/mysql/manifests/latest'
5. It worked! And now, I get the following:
{"manifests":[{"annotations":{"com.docker.official-images.bashbrew.arch":"amd64","org.opencontainers.image.base.digest":"sha256:c0de6b893513115fbaa4e643f8d2893630eb5362943c596717a7b6a78aef69ec","org.opencontainers.image.base.name":"oraclelinux:9-slim","org.opencontainers.image.created":"2024-05-09T23:51:50Z","org.opencontainers.image.revision":"a15b34a032f48089ee7b02d307d8f89a96b3bb76","org.opencontainers.image.source":"https:\/\/github.com\/docker-library\/mysql.git#a15b34a032f48089ee7b02d307d8f89a96b3bb76:8.4","org.opencontainers.image.url":"https:\/\/hub.docker.com\/_\/mysql","org.opencontainers.image.version":"8.4.0"},"digest":"sha256:e193c837211ee976fd9b638740a0264eff9b1d602d3affed7938ddbde7ff3035","mediaType":"application\/vnd.oci.image.manifest.v1+json","platform":{"architecture":"amd64","os":"linux"},"size":2855},{"annotations":{"com.docker.official-images.bashbrew.arch":"amd64","vnd.docker.reference.digest":"sha256:e193c837211ee976fd9b638740a0264eff9b1d602d3affed7938ddbde7ff3035","vnd.docker.reference.type":"attestation-manifest"},"digest":"sha256:0e58aef73dd0a7b787428875aa2b64600c048aa5e55e8529b0bb716d4678ede9","mediaType":"application\/vnd.oci.image.manifest.v1+json","platform":{"architecture":"unknown","os":"unknown"},"size":842},{"annotations":{"com.docker.official-images.bashbrew.arch":"arm64v8","org.opencontainers.image.base.digest":"sha256:5a8494823c700a8003ab2bb87220d0389362a33fd9086f20be1172925bca0825","org.opencontainers.image.base.name":"oraclelinux:9-slim","org.opencontainers.image.created":"2024-05-09T23:37:48Z","org.opencontainers.image.revision":"a15b34a032f48089ee7b02d307d8f89a96b3bb76","org.opencontainers.image.source":"https:\/\/github.com\/docker-library\/mysql.git#a15b34a032f48089ee7b02d307d8f89a96b3bb76:8.4","org.opencontainers.image.url":"https:\/\/hub.docker.com\/_\/mysql","org.opencontainers.image.version":"8.4.0"},"digest":"sha256:79bf3d05d420c85d21702dcde491ec36be535c79200cc83f1298dd838a6f1dad","mediaType":"application\/vnd.oci.image.manifest.v1+json","platform":{"architecture":"arm64","os":"linux","variant":"v8"},"size":2857},{"annotations":{"com.docker.official-images.bashbrew.arch":"arm64v8","vnd.docker.reference.digest":"sha256:79bf3d05d420c85d21702dcde491ec36be535c79200cc83f1298dd838a6f1dad","vnd.docker.reference.type":"attestation-manifest"},"digest":"sha256:ad0644d4ea4c2291a9c3990d2e0373112058f7e7d8a3da4865c0c079b9c878cd","mediaType":"application\/vnd.oci.image.manifest.v1+json","platform":{"architecture":"unknown","os":"unknown"},"size":842}],"mediaType":"application\/vnd.oci.image.index.v1+json","schemaVersion":2}

4. So, I get a good response from the curl. Now, I checked the proxy and it cached the image.

HOWEVER, the docker login acts strangely. I login to the localhost docker proxy with username and password supposedly successfully, then “docker logout” and it always says, “Removing login credentials for https://index.docker.io/v1/”. I was expecting it to say, " Removing login credentials for http://localhost:8082". This is very odd!

So, the docker pull still fails to cache image in Nexus 3 docker proxy.

[Update] I just found this issue: Docker is not passing auth informations when pulling from a mirror registry · Issue #30880 · moby/moby · GitHub. Sound like it’s a problem. What do you think?

BTW, this did not work for me as per issue …

Docker version 20.10.12
Nexus version 3.62.0-01
In settings repository enable Allow anonymous docker pull
Then need go to settings -> realms and enable Docker Bearer Token Realm. This help me with problem:
Error response from daemon: unauthorized: authentication required

I disabled the anonymous and I got this error in the log file:

 dockerd[429543]: time="2024-05-16T22:30:11.871726625-05:00" level=info msg="Attempting next endpoint for pull after error: Head \"http://localhost:8082/v2/library/alpine/manifests/3.17\": no basic auth credentials"

So, if you have an ideas. Regards -Alan

[rimelek (Ákos Takács)]

Thank you fo the more detailed description of the issue.

@aguedeney

Where would I need to look to see the docker daemon log file(s)?

It seems you found it eventually.

@aguedeney

But, unfortunately, the docker pull completely ignored it and failed to pull from Nexus 3’s Docker proxy and went directly to Dockerhub.

You mean you are sure the image was already in the local registry but regardless of that, docker pulled the image again from Docker Hub? That would mean the pull through cache registry works, but allows you to pull an image from the hub even when you don’t have permission to the pull thorugh cache registry.

Since I never configured a pull through cache registry before, I’m not sure how it is supposed to handle the auth process, but it would be weird indeed and I don’t know how it could happen.

@aguedeney

Head \"http://localhost:8082/v2/library/mysql/manifests/latest\": unauthorized: access to the requested resource is not authorized"

When a registry says you are not authorized to access an image, sometimes it is because the image cannot be found in the registry and I believe the registry responds the same way when you don’t have permission to an image or when the image does not exist. So maybe you are right and there is a problem with using the credentials, or maybe an error happened and the registry tried to find the image locally.

@aguedeney

then “docker logout” and it always says, “Removing login credentials for https://index.docker.io/v1/”. I was expecting it to say, " Removing login credentials for http://localhost:8082". This is very odd!

The logout command requires the server as parameter the same way as docker login. You could log in to multiple registries and the logout command will log you out from only one server. If you don’t pass the server as argument, it will fall back to the default, which is Docker Hub. So if you just ran docker logout without parameters, there is nothing odd in it.

Sorry, but I didn’t read the whole issue on GitHub, I just saw they referred to a Red Hat implementation of Docker. I understand the solution you found didn’t work for you, but as I didn’t need to configure a pull through cache registry yet, I’m affraid, you will need to wait for someone who speaks from experience. Hopefully it will happen sooner than I will have time to try it

[aguedeney]

None-the-less, it answered some of my questions. Thank you.