Docker-compose pull command fails with pull access denied

Docker Hub
,
1

I am using gitlab private registry. As part of the CI/CD job. I login with the CI_REGISTRY_USER as well a token with all the access enabled to gitlab. (tried these two logins instructions separately)

echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY

docker login -u "gitlab-deploy-token" -p "xxxxxxxxxx" $CI_REGISTRY

then start by taking down the currently running service containers and followed by an image pull.

docker-compose -f docker-compose.dev.yaml down --remove-orphans
docker-compose -f docker-compose.dev.yaml --verbose pull
docker-compose -f docker-compose.dev.yaml up -d --no-build --abort-on-container-exit

If I remove the pull instruction everything works correctly. Here is the output of the docker-compose pull. I tried the above instructions with token that is given by Gitlab as well as one created by me (see above). It still doesn’t work. It always gives me pull access denied.

Can someone tell me what is going wrong here ?

docker-compose -f docker-compose.dev.yaml --verbose pull
compose.config.config.find: Using configuration files: ./docker-compose.dev.yaml
docker.utils.config.find_config_file: Trying paths: ['/home/gitlab-runner/.docker/config.json', '/home/gitlab-runner/.dockercfg']
docker.utils.config.find_config_file: Found file at path: /home/gitlab-runner/.docker/config.json
docker.auth.load_config: Found 'auths' section
docker.auth.parse_auth: Found entry (registry='gitlab.xxxx.fi:4567', username='gitlab-deploy-token')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/version HTTP/1.1" 200 865
compose.cli.command.get_client: docker-compose version 1.25.4, build 8d51620a
docker-py version: 4.1.0
CPython version: 3.7.5
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
compose.cli.command.get_client: Docker base_url: http+docker://localhost
compose.cli.command.get_client: Docker version: Platform={'Name': 'Docker Engine - Community'}, Components=[{'Name': 'Engine', 'Version': '19.03.6', 'Details': {'ApiVersion': '1.40', 'Arch': 'amd64', 'BuildTime': '2020-02-13T01:26:21.000000000+00:00', 'Experimental': 'false', 'GitCommit': '369ce74a3c', 'GoVersion': 'go1.12.16', 'KernelVersion': '4.15.0-106-generic', 'MinAPIVersion': '1.12', 'Os': 'linux'}}, {'Name': 'containerd', 'Version': '1.2.10', 'Details': {'GitCommit': 'b34a5c8af56e510852c35414db4c1f4fa6172339'}}, {'Name': 'runc', 'Version': '1.0.0-rc8+dev', 'Details': {'GitCommit': '3e425f80a8c931f88e6d94a8c831b9d5aa481657'}}, {'Name': 'docker-init', 'Version': '0.18.0', 'Details': {'GitCommit': 'fec3683'}}], Version=19.03.6, ApiVersion=1.40, MinAPIVersion=1.12, GitCommit=369ce74a3c, GoVersion=go1.12.16, Os=linux, Arch=amd64, KernelVersion=4.15.0-106-generic, BuildTime=2020-02-13T01:26:21.000000000+00:00
compose.cli.verbose_proxy.proxy_callable: docker inspect_network <- ('docker_backend')
urllib3.connectionpool._make_request: http://localhost:None "GET /v1.38/networks/docker_backend HTTP/1.1" 404 52
Pulling postgresql             ... 
Pulling core-application ... 
Pulling frontend         ... 
compose.parallel.feed_queue: Pending: {<Service: postgresql>, <Service: frontend>, <Service: core-application>}
compose.parallel.feed_queue: Starting producer thread for <Service: postgresql>
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/custom-postgres', tag='latest', stream=True, platform=None)
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
compose.parallel.feed_queue: Starting producer thread for <Service: frontend>
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests', tag='latest', stream=True, platform=None)
compose.parallel.feed_queue: Starting producer thread for <Service: core-application>
docker.auth.get_config_header: Looking for auth config
compose.cli.verbose_proxy.proxy_callable: docker pull <- ('gitlab.xxxx.fi:4567/webapp/core-application/devops-pipeline-tests', tag='latest', stream=True, platform=None)
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Looking for auth config
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.resolve_authconfig: Looking for auth entry for 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
docker.auth.resolve_authconfig: Found 'gitlab.xxxx.fi:4567'
docker.auth.get_config_header: Found auth config
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Fcore-application%2Fdevops-pipeline-tests HTTP/1.1" 404 216
compose.parallel.parallel_execute_iter: Failed: <Service: core-application>
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Fcustom-postgres HTTP/1.1" 404 193
compose.parallel.parallel_execute_iter: Failed: <Service: postgresql>
compose.parallel.feed_queue: Pending: set()
urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Ffrontend%2Fdevops-pipeline-tests HTTP/1.1" 404 208
compose.parallel.parallel_execute_iter: Failed: <Service: frontend>
compose.parallel.feed_queue: Pending: set()
ERROR: for core-application  pull access denied for gitlab.xxxx.fi:4567/webapp/core-application/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for postgresql  pull access denied for gitlab.xxxx.fi:4567/webapp/custom-postgres, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
ERROR: for frontend  pull access denied for gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
compose.cli.errors.log_api_error: pull access denied for gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests, repository does not exist or may require 'docker login': denied: requested access to the resource is denied
2

Are you able to do docker pull gitlab.xxxx.fi:4567/webapp/frontend/devops-pipeline-tests from the cli? If this works, it should work from inside docker-compose as well. Is it by intention that no tag is provided for the image? By convention it will replace an emtpy tag with latest, though, the tag is not automaticly created when building images.

Either CI_REGISTRY is not gitlab.xxxx.fi:4567, the user lacks permissions to access the repo or the latest tag is simply missing.

3

@meyay I can pull from my local machine. That has nothing to do with the gitlab job running in a gitlab-runner VM in a cloud. The instruction without the explicit pull works. I have created a deploy token with all permissions and it still doesn’t work. I even logged in to that gitlab-runner machine and pull the images explicitly and it works.

I can see from the logs that the docker-compose is trying to access the localhost for some reason.

urllib3.connectionpool._make_request: http://localhost:None "POST /v1.38/images/create?tag=latest&fromImage=gitlab.xxxx.fi%3A4567%2Fwebapp%2Fcore-application%2Fdevops-pipeline-tests HTTP/1.1" 404 216
all the images has latest set in the gitlab. I just didn’t give the :latest tag in the docker-compose file. (as it is the default)