We are using private registry to host our own application images. Total of 4 dockers host engines. 1 swarm manager and 3 workers. On Manager we have created the image using private registry and after that used docker service create command to create the service with 4 replicas. Unfortunately the workers were not able to pull the image from private registry and while looking at the logs it say,
If we run a pull command within swarm worker, then swarm will be able to use that image and the container is running properly… The issue is the image is not available with in local and it was supposed to be pulled up from the registry. Please suggest how to fix this.
It’s not the issue with image availability on private registry. If I execute docker run command on swarm worker directly it’s working fine. I am able to create the container. Only issue is when the work was distributed by swarm manager. The swarm setup holds good.
I have 1 swarm manager and 3 swarm workers.
Couple of scenarios:
take an example like haproxy Image is available on all swarm workers. Run docker service create with 5 replicas, then it was able to distribute the work to 3 swarm workers and 5 containers were created and started properly.
Take an example like nginx image which is not available on swarm workers. The images was available only on private registry and the swarm service create command has the information of private registry along with image name. Run docker service create command to create 5 replicas. the work was distributed I can see that with docker service tasks command but the containers were not created and when I check the logs it’s saying the image was not found. If I execute the run command on the swarm worker directly it’s working fine. But this doesn’t hold good. I will be changing the images and swarm worker should be able to pull the images from private registry instead of loading the image on each worker.
@chkrishna what does your “docker service create” command look like.
I’m trying the same thing and cannot get this to work. My private registry requires basic auth. Before issuing the “docker service create” command I successfully did a docker login against my registry
Jul 20 15:11:05 myhost dockerd: time="2016-07-20T15:11:05.733316229Z" level=error msg="Attempting next endpoint for pull after error: Get https://dockerhub-test.mydomain.com/v2/app1/app1-authorization/manifests/latest: no basic auth credentials"
Does anyone have this working??
I see the same behavior, it only works if you go and manually pull the images on each swarm worker…
Finally after days of struggle, it’s working fine,.
we are using Docker version 1.12.0-rc4, build e4a0dbc, experimental
Previously I didn’t use the login command first and directly tried with registry-auth for create command.
Currently I am issuing following 3 commands.
Created N/w using,
docker network create -d overlay mynetwork
After that it was able to successfully pull the image from private registry on all swarm nodes and started the servers. But one more problem that I am facing is specifying the random ports for each container to forward from host. I tried --publish option while creating the service, but it didn’t help.
so, its not clear to me, what does the “registry-auth” parameter actually do? If I do a docker login [myreg] on each swarm node, is this still necessary?
registry-auth option with docker service create, informs the manager to forward the registry credentials to swarm workers.
As per the documentation,
–registry-auth Send registry authentication details to Swarm agents
you don’t need to login in each swarm node individually. Think about bringing up new swarm workers. it should pull the images automatically when we scale the service and distributed to new swarm workers.
I see that you use MYUSERNAME and MYPASSWORD… and that probably works pretty well, however, if you use gitlab CI/CD tokens and docker swarm it’s probably going to fail when the CI token expires. Second I have not heard anyone describe the TTL for --with-registry-auth
I think I just ran into the specific issue of GitLab CI tokens expiring. I added a worker to my swarm and immediately a global service triggered a load of rejected docker pull requests. As an aside, the endless rapidly repeating attempts to pull an image that doesn’t exist or that the server doesn’t have access to is a really annoying feature of Swarm Mode. It can (and for me has on several occasions) cause your server to be blocked by the GitLab registry for a day or more simply because a token expired or a mistake in your build script caused it to deploy without publishing the required container image first.
I’ve just come across the option of using Deploy Tokens rather than Job Tokens in GitLab CI, which I think helps because it gives you control over the TTL (you can even create one without a TTL, although that seems quite insecure for production systems). I haven’t tested it yet, but I presume if you make sure to redeploy your stack/service through CI immediately after each rotation of the Deploy Token, that will push the new token to the swarm and prevent any issues with expired CI tokens.