Service task can't pull from private registry

[bouga]

Hey!

I’m running a service on a Swarm cluster, thanks to docker stack deploy --with-registry-auth and this compose file:

version: "3.1"
services:
  builder-consumer:
    image: us.gcr.io/my-gcloud-project/my/image:123
    stop_grace_period: 30m
    volumes:
      - [...]
    environment:
      - [...]
    deploy:
      mode: global
      placement:
        constraints:
          - node.role == worker
    secrets:
      - [...]
secrets:
  [...]

This works fine when I deploy, but when I add a worker node to the swarm later on, the new worker can’t pull the image required to run the task. The system logs report this:

level=error msg="Not continuing with pull after error: denied: Permission denied for \123\" from request \"/v2/my-gcloud-project/my/image/manifests/123\". "

level=info msg="Translating \"denied: Permission denied for \\\"123\\\" from request \\\"/v2/my-gcloud-project/my/image/manifests/123\\\". \" to \"repository us.gcr.io/my-gcloud-project/my/image not found: does not exist or no pull access\""

level=error msg="pulling image failed" error="repository us.gcr.io/my-gcloud-project/my/image not found: does not exist or no pull access" module="node/agent/taskmanager" node.id=... service.id=... task.id=...

level=error msg="fatal task error" error="No such image: us.gcr.io/my-gcloud-project/my/image:123@sha256:..." module="node/agent/taskmanager" node.id=... service.id=... task.id=...

However, when I manually run docker pull on that machine, it works fine, since every machine in the cluster is authenticated to my private Google Registry, thanks to docker login.

Thus my questions are:

• Why can’t the added worker pull from the private registry?
• What does --with-registry-auth do exactly?

Thanks a lot

EDIT: the nodes are running Ubuntu 16.04.2 LTS and the Docker version is:

Server:
 Version:      17.04.0-ce
 API version:  1.28 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   4845c56
 Built:        Mon Apr  3 18:07:42 2017
 OS/Arch:      linux/amd64
 Experimental: false

[dskwirut]

I’m having the same issue too.

On the individual swarm nodes, I can do a docker login and docker pull from my registry just fine.

But if I do:

docker service create --replicas 1 --publish 8080:8000 --name newtest2 --with-registry-auth docker.privateregistry.com/newtest2

I start getting messages saying:

level=error msg="Attempting next endpoint for pull after error: unknown: Authentication is required"
level=error msg=“pulling image failed” error=“Get: unknown: Authentication is required” module=taskmanager
level=error msg=“fatal task error” error="No such image:

Is it not possible to use swarm with private registry?

[bouga]

Does anyone know a solution to this issue?

[asmallfurrywombat]

As near as i can determine, docker stack deploy has problems reliably publishing service ports declared using a yml file. If you create the registry service using the docker cli then it should work. Perhaps this helps:
https://lostechies.com/gabrielschenker/2016/09/05/docker-and-swarm-mode-part-1/

[bouga]

I don’t manually run my private registry, I use Google Container Registry, so it’s transparently managed by Google.

I doubt it’s a problem with the registry istelf, since when I manually run docker pull on the new worker node, it successfully pulls the image from the GCR.

What I don’t understand is: why can’t my service pull that private image itself?

Am I doing something wrong with my authentication configuration? Or is there a bug in docker swarm or the registry-auth?

[asmallfurrywombat]

some people are using registry authentication but don’t use “docker login” followed by “–with-registry-auth”.

[bouga]

Any idea how they manage to do that?

[bouga]

When a worker node pulls an image, does it pull it is as a specific user?

Which user should be authenticated to my registry to make it work?

[abhishekgaloda]

I had to do 2 things to deploy docker images from private repo in a docker Swarm

• docker login
  followed by something like

• docker login -u abhishek -p password registry.hub.Docker.com/myproject && Docker stack deploy -c Docker-swarm.yml test --with-registry-auth

Read below link for more details and example

Installing Docker Images from private repositories in Docker Swarm - Little Big...

[davidpccode]

It’s 2021… I’m facing same problem!!

Docker Engine 20.10
My repository: Docker Hub

I followed all instructions and nothing. I also execute the command (with my data of course)

docker login -u my-username -p my-password registry.hub.Docker.com/my-username && Docker stack deploy -c Docker-swarm.yml mystack-name --with-registry-auth

I’ve been researching and testing everything for more than 12 hours and I haven’t been able to.

When I deploy “docker stack deploy” replicas can’t be created because
“No such image: user/image:tag”

Any solution? I would really appreciate it
Thanks!

[toyfrog (TF)]

Hi @bouga ,

Are you adding the node without rerunning the docker stack deploy command?
If you rerun docker stack deploy after adding a new node, do things work as you expected?

It could be that the --with-registry-auth flag pushes credentials to the nodes that exist in the swarm at the time you run the command, but the auth info may not propagate to new nodes as they join. I am not sure if this is expected behavior, but if rerunning docker stack deploy again after adding a new node, then this is likely what is going on in the background.