krishnaku
January 18, 2022, 4:59pm
1
Hello,
I have a two stage docker build process that creates a base image which has all the static dependencies of the app, and a second stage that used this base image to install the latest application code.
This process has been working fine for several years now, and started to fail after I upgraded from Docker 4.2.0 to Docker 4.4.2. I am not sure what changed in between these two releases, but searching on SO/Google etc did not turn up any relevant answers. so I am posting here to see if there might be some help available.
The error I am seeing is this:
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed**
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
But it is throwing this error when trying to look up the image metadata for the base image that It just finished building in the previous step. This image is local and not pushed to any registry, but it looks like the build process is trying to look this up from the docker.io registry.
NOTE: since I am a first time user, this forum seems to be rejecting my post because it has more than two links. So I am unable to post the actual logs which it thinks contains links,
Any help would be appreciated.
Krishna
rimelek
January 24, 2022, 7:26pm
2
You shouldn’t share logs without using the </> button (code block or as the forum calls it “preformatted text”). Your logs could contain other characters too like * which would make a bullet list “italic” text.
example
this is the log with links : docker.io forums.docker.com
Those logs would be useful. Docker would not look for metadata online unless you built image got a different name than you expected or it wasn’t built at all in the previous step.
krishnaku
January 24, 2022, 8:36pm
3
Thanks Akos… my bad for not figuring out the code block issue
I was able to get everything to build just by turning build_kit off. So that’s how I am working around the issue for now. So I suspect this is a bug of some sort for sure.
I’ll try and reproduce this with build_kit=1 and send you the logs.
rimelek
January 24, 2022, 8:48pm
4
Build kit again… This is what I was talking about to @meyay in an other topic.
I know a little bit more about buildkit since then but I don’t know why it would cause this issue. When you enable buildkit, your images are built using runc directly. but it should not check metadata online of a local image either. When I am done with my docker build tutorial without buildkit, I should start investigating more how buildkit really works.
Your logs may help to understand your case if you can reproduce it.
krishnaku
January 25, 2022, 10:16pm
5
Here is the complete build log.
Pulling registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Building package base image: polaris-utils-base with Builder Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and Release Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#1 [internal] load build definition from Dockerfile
#1 sha256:72a2373a5f034babd468efd358fd3fcb1e7ae60c6b70ddd9fbd89d59c4d6ce2a
#1 transferring dockerfile: 2.08kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:cf164168c4c6516cb0dd2d1d1b65c047c81ed6983a7ed0714f424c0725568fc0
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#3 sha256:d2bd26b30afc04ce4f9781ab6637055fcedd793ac63d48a928eb1af5834b793a
#3 ...
#4 [auth] polaris-common/polaris-build/polaris-build/polaris-python-build-base:pull token for registry.gitlab.com
#4 sha256:d8c2891d3d6648146009781d4dc867f1165be6d5eed53bef803fa59c47930f11
#4 DONE 0.0s
#3 [internal] load metadata for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#3 sha256:d2bd26b30afc04ce4f9781ab6637055fcedd793ac63d48a928eb1af5834b793a
#3 DONE 1.9s
#5 [stage-0 1/7] FROM registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest@sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
#5 sha256:191f5d4a5139de326cb13ee047be13f1576c7c90908220920a46141e75864ec6
#5 DONE 0.0s
#6 [internal] load build context
#6 sha256:e1d5bf1134e2e0a47987ff78f0acbe5bed6b64d3fc726ed5132e67dc76bc85e4
#6 transferring context: 12.41kB 0.0s done
#6 DONE 0.0s
#13 [stage-1 2/4] COPY --from=0 /project/wheels /tmp/build/wheels/
#13 sha256:4c1857940cc31b91fc1d2bbe70e3bf1197d3ed2f9dd494f558852c1f2495feb3
#13 CACHED
#7 [stage-0 2/7] COPY *requirements.txt polaris-build/system-pip-constraints.txt /tmp/build/
#7 sha256:a4d464168708cfda826cbeb8d0969a186baad22542e371ccfd10a6a1de00cf6c
#7 CACHED
#12 [stage-0 7/7] RUN pip freeze | grep -v polaris > /project/requirements.txt || true && pip freeze | grep polaris > /project/polaris-requirements.txt || true
#12 sha256:8c5ce7233bc9b8ed77b4340f4a7e81650e399c091f7cbb67e2473b828ffe6d5c
#12 CACHED
#11 [stage-0 6/7] RUN mkdir -p /project/wheels && ./install.sh --dependencies-only
#11 sha256:d3e247cde9c8be2beb13b269cce5b1ec951118ae829d62ecbf1d019f889a22a8
#11 CACHED
#9 [stage-0 4/7] COPY . /project/
#9 sha256:2dfc6faba2b315b4a39c182883027d7bc62755cf4cfbb03a3197b8dfc3a77cc4
#9 CACHED
#10 [stage-0 5/7] WORKDIR /project
#10 sha256:fb6f3cf7d501af7f149e296d8cf45a8794bb64e19fbdd6794de567ce21edf541
#10 CACHED
#8 [stage-0 3/7] RUN touch /tmp/build/all-requirements.txt && cat /tmp/build/all-requirements.txt | grep -v polaris > /tmp/build/dependencies.txt || true && pip install -r /tmp/build/dependencies.txt -c /tmp/build/system-pip-constraints.txt && rm -rf /tmp/build
#8 sha256:5675b92fba82f8b91e405df3e0794fe133c0c3fcbbb2f94068c060489ebb156c
#8 CACHED
#14 [stage-1 3/4] COPY --from=0 /project/requirements.txt /project/polaris-requirements.txt /tmp/build/
#14 sha256:145586e0047f38264fe53bd733457fd985652b577f13a4aa605dd05c1c002409
#14 CACHED
#15 [stage-1 4/4] RUN cat /tmp/build/*requirements.txt | xargs -I '?' pip install --no-index --find-links /tmp/build/wheels ? && rm -rf /tmp/build
#15 sha256:9f8a482e54d6ff8ffe14bd1f873cbb6d59fc77b2a4e5d29d9c351a97e4ad40a0
#15 CACHED
#16 exporting to image
#16 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#16 exporting layers done
#16 writing image sha256:2b416b81dab0fd8b5004f44495b7c4347c15cef0881b8dead5df2c4d357151f5 done
#16 naming to docker.io/library/polaris-utils-base done
#16 DONE 0.0s
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building package image: polaris-utils
#1 [internal] load build definition from Dockerfile
#1 sha256:ee306a68d614211594e97628b6c6b1fcf290338ee4e55ef2ad032bd355ec017e
#1 transferring dockerfile: 758B done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 sha256:dfbfc751fd0d76f70a4f499f66c96d6ef2fa9b89d725abf07dc868e14dcd80e4
#2 transferring context: 2B done
#2 DONE 0.0s
#3 [internal] load metadata for docker.io/library/polaris-utils-base:latest
#3 sha256:4ac34a971b9e19a3ab5761a049fa8fa1f8c27e1cf07880f0fedb61795e3d972b
#3 ...
#4 [auth] library/polaris-utils-base:pull token for registry-1.docker.io
#4 sha256:37ec73dfef4fa6bbe5ba066bbe85ae6ef623005d3e51ae4f8ed869526e80224a
#4 DONE 0.0s
#3 [internal] load metadata for docker.io/library/polaris-utils-base:latest
#3 sha256:4ac34a971b9e19a3ab5761a049fa8fa1f8c27e1cf07880f0fedb61795e3d972b
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
> [internal] load metadata for docker.io/library/polaris-utils-base:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Error: build package failed.
Here is the log with DOCKER_BUILDKIT=0
Krishnas-Mac-Mini-M1:polaris-utils krishna$ DOCKER_BUILDKIT=0 package build
Pulling registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Building package base image: polaris-utils-base with Builder Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and Release Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Sending build context to Docker daemon 519.9kB
Step 1/13 : ARG BUILDER_IMAGE
Step 2/13 : ARG RELEASE_IMAGE
Step 3/13 : FROM ${BUILDER_IMAGE}
---> c14a1fddf37c
Step 4/13 : COPY *requirements.txt polaris-build/system-pip-constraints.txt /tmp/build/
---> Using cache
---> 4700e7a74924
Step 5/13 : RUN touch /tmp/build/all-requirements.txt && cat /tmp/build/all-requirements.txt | grep -v polaris > /tmp/build/dependencies.txt || true && pip install -r /tmp/build/dependencies.txt -c /tmp/build/system-pip-constraints.txt && rm -rf /tmp/build
---> Using cache
---> 34f8729066ea
Step 6/13 : COPY . /project/
---> Using cache
---> fba3d292f3eb
Step 7/13 : WORKDIR /project
---> Using cache
---> 9c21390dc85e
Step 8/13 : RUN mkdir -p /project/wheels && ./install.sh --dependencies-only
---> Using cache
---> 446df7f62b85
Step 9/13 : RUN pip freeze | grep -v polaris > /project/requirements.txt || true && pip freeze | grep polaris > /project/polaris-requirements.txt || true
---> Using cache
---> 72be2b1f6872
Step 10/13 : FROM ${RELEASE_IMAGE}
---> c14a1fddf37c
Step 11/13 : COPY --from=0 /project/wheels /tmp/build/wheels/
---> Using cache
---> a6db9b561cca
Step 12/13 : COPY --from=0 /project/requirements.txt /project/polaris-requirements.txt /tmp/build/
---> Using cache
---> 032adeadb813
Step 13/13 : RUN cat /tmp/build/*requirements.txt | xargs -I '?' pip install --no-index --find-links /tmp/build/wheels ? && rm -rf /tmp/build
---> Using cache
---> f789f46bc7ac
Successfully built f789f46bc7ac
Successfully tagged polaris-utils-base:latest
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building package image: polaris-utils
Sending build context to Docker daemon 519.9kB
Step 1/12 : ARG PACKAGE_BASE_IMAGE
Step 2/12 : FROM ${PACKAGE_BASE_IMAGE}
---> f789f46bc7ac
Step 3/12 : ARG ALEMBIC_ROOT
---> Using cache
---> 5f22825b2355
Step 4/12 : ENV ALEMBIC_ROOT ${ALEMBIC_ROOT:-.}
---> Using cache
---> 10bbe751ad8a
Step 5/12 : COPY . /project/
---> Using cache
---> 050fc5cde3fa
Step 6/12 : WORKDIR /project
---> Using cache
---> 657b59bbce96
Step 7/12 : RUN ./install.sh && rm -rf /project/*
---> Using cache
---> 818ff9090dbd
Step 8/12 : COPY conftest.py pytest.ini /project/
---> Using cache
---> 704fbc684559
Step 9/12 : COPY test /project/test/
---> Using cache
---> 990181fc84a8
Step 10/12 : COPY polaris-build/bin/* bin/* /usr/local/bin/
---> Using cache
---> 69dc18c78067
Step 11/12 : COPY polaris-build/ignore-missing ${ALEMBIC_ROOT}/alembic.ini* /package_migrations/
---> Using cache
---> b8ec45a81776
Step 12/12 : COPY polaris-build/ignore-missing ${ALEMBIC_ROOT}/migrations* /package_migrations/migrations/
---> Using cache
---> 2f027be3d6ae
Successfully built 2f027be3d6ae
Successfully tagged polaris-utils:latest
Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Executing post build steps for python package..
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Please note: all-requirements.txt has been updated. Please commit this change.
krishnaku
January 25, 2022, 10:18pm
6
Also I can confirm that this is not a 4.4.2 issue. I was able to reproduce this with 4.2.0 as well. Another bit of perhaps important info: this build is running on a new Mac with the M1 chip, whereas all my other machines are running the intel chip.
rimelek
January 31, 2022, 9:02pm
7
Thank you for the logs! Sorry I am tring to catch up with the unread posts.
After I read only your previous post and searched for the error message, I would have asked if you had a Mac with the M1 chip, because it looks like it was not really compatible with Docker Compose, especually with buildkit.
opened 02:33PM - 27 Jul 21 UTC
closed 10:26PM - 25 Jan 23 UTC
kind/bug
Docker Compose V1
Hi!
## Description of the issue
I recently updated docker and after update… it seems that Dockerfile cannot use local images anymore. I have this instance:
Which is later used by another service in Dockerfile using `FROM services_develop:staging`. Prior update it was working just fine, now only option seems to be to create a local registry. Is this really intended?
## Context information (for bug reports)
**Output of `docker-compose version`**
```
docker-compose version 1.29.2, build 5becea4c
docker-py version: 5.0.0
CPython version: 3.9.0
OpenSSL version: OpenSSL 1.1.1h 22 Sep 2020
```
/!\ If `docker-compose version` reports v2.0...` you're using the wrong repository, please report issues on [Compose-CLI](https://github.com/docker/compose-cli/issues/new)
**Output of `docker version`**
```
Client:
Cloud integration: 1.0.17
Version: 20.10.7
API version: 1.41
Go version: go1.16.4
Git commit: f0df350
Built: Wed Jun 2 11:56:23 2021
OS/Arch: darwin/arm64
Context: default
Experimental: true
Server: Docker Engine - Community
Engine:
Version: 20.10.7
API version: 1.41 (minimum version 1.12)
Go version: go1.13.15
Git commit: b0f5bc3
Built: Wed Jun 2 11:55:36 2021
OS/Arch: linux/arm64
Experimental: false
containerd:
Version: 1.4.6
GitCommit: d71fcd7d8303cbf684402823e425e9dd2e99285d
runc:
Version: 1.0.0-rc95
GitCommit: b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
docker-init:
Version: 0.19.0
GitCommit: de40ad0
```
**Output of `docker-compose config`**
(Make sure to add the relevant `-f` and other flags)
```
networks:
default:
driver: bridge
pm_default:
external: true
name: pm_default
services:
build:
build:
args:
SRC_MOUNT_PATH: /srv/sites/services
context: /Users/xxx/Documents/Projects/sites/yyy/services/docker/build
depends_on:
develop:
condition: service_started
environment:
DEPLOYMENT_PROD_HOSTS: ''
DEPLOYMENT_PROD_KEY_FILENAME: ''
DEPLOYMENT_PROD_KEY_PASSPHRASE: ''
DEPLOYMENT_PROD_LOCAL_SCRIPTS_PATH: ''
DEPLOYMENT_PROD_REMOTE_SCRIPTS_PATH: ''
DEPLOYMENT_PROD_RUNTIME_PATH: /srv/sites/yyy/services
DEPLOYMENT_PROD_TEST_RUNTIME_PATH: /srv/sites/yyy/testing.services
DEPLOYMENT_PROD_TMP_PATH: /tmp
DEPLOYMENT_PROD_USERNAME: root
RUNTIME_PATH: /srv/sites/services/pm_services
SRC_MOUNT_PATH: /srv/sites/services
image: services_build:staging
networks:
default: null
volumes:
- /Users/xxx/Documents/Projects/sites/yyy/services:/srv/sites/services:cached
develop:
build:
args:
SRC_MOUNT_PATH: /srv/sites/services
context: /Users/xxx/Documents/Projects/sites/yyy/services/docker/develop
depends_on:
redisdb:
condition: service_started
environment:
DEPLOYMENT_PROD_HOSTS: ''
DEPLOYMENT_PROD_KEY_FILENAME: ''
DEPLOYMENT_PROD_KEY_PASSPHRASE: ''
DEPLOYMENT_PROD_LOCAL_SCRIPTS_PATH: ''
DEPLOYMENT_PROD_REMOTE_SCRIPTS_PATH: ''
DEPLOYMENT_PROD_RUNTIME_PATH: /srv/sites/yyy/services
DEPLOYMENT_PROD_TEST_RUNTIME_PATH: /srv/sites/yyy/testing.services
DEPLOYMENT_PROD_TMP_PATH: /tmp
DEPLOYMENT_PROD_USERNAME: root
RUNTIME_PATH: /srv/sites/services/pm_services
SRC_MOUNT_PATH: /srv/sites/services
image: services_develop:staging
networks:
default: null
pm_default: null
ports:
- published: 4100
target: 4100
- published: 4200
target: 4110
volumes:
- /Users/xxx/Documents/Projects/sites/yyy/services:/srv/sites/services:cached
redisdb:
image: redis:alpine
version: '3.9'
```
## Steps to reproduce the issue
1. `docker-compose up --build build`
### Observed result
Fails to run
### Expected result
Container is run
### Stacktrace / full error message
```
#2 [services_build:staging internal] load build definition from Dockerfile
#2 sha256:904e5663af5547b640ab36250d9303488863d83cb6ca4c0111b17700a843eec9
#2 transferring dockerfile: 337B done
#2 DONE 0.0s
#1 [services_develop:staging internal] load build definition from Dockerfile
#1 sha256:d8973824e1cc852c8a3ba858f1c282f2943a049f4c86b2cc88e4ecfb20912cb3
#1 transferring dockerfile: 860B done
#1 DONE 0.0s
#3 [services_develop:staging internal] load .dockerignore
#3 sha256:a577a2d1f67337b83d0055cf11cf3d65ec16157e002120d74886257da7875479
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [services_develop:staging internal] load metadata for docker.io/library/debian:stable
#4 sha256:03abc9f10f935e7137bbfe987d86a0879bf4bcb2ca500f3919314c6aa058f2f8
#4 ...
#5 [services_build:staging internal] load .dockerignore
#5 sha256:a3a79252ae83d732c535c559ffb437f9be6ae841462892d5b0c084a22f5ebf2b
#5 transferring context: 2B done
#5 DONE 0.0s
#6 [services_build:staging internal] load metadata for docker.io/library/services_develop:staging
#6 sha256:0aabca4baec97ad783f7b602174718f61033d1eb5568a3e962ea3e70e39858bb
#6 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
#4 [services_develop:staging internal] load metadata for docker.io/library/debian:stable
#4 sha256:03abc9f10f935e7137bbfe987d86a0879bf4bcb2ca500f3919314c6aa058f2f8
#4 CANCELED
------
> [services_build:staging internal] load metadata for docker.io/library/services_develop:staging:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
```
## Additional information
Macbook Pro M1, MacOS 11.4.
I tried to try your image
krishnaku
January 31, 2022, 9:34pm
8
I am using Docker Compose fine on this machine (my entire dev stack is built and run using this) with buildit turned off. My point is that the image is never published to Docker Hub. I actually don’t use docker hub at all - my registries are on GitLab. But in this case, there is no real registry involved anywhere, I am creating a local image in one step of the build and using that local image in the next step of the build process and for some reason it seems to be looking on Docker Hub for that intermediate image.
Anyway, I hope this can get fixed, in the meantime, I have moved all my actual production builds to a Linux machine and so I am not blocked by this anymore. But it still seems like an important bug to fix for folks who are using Docker Compose and newer Macs to build images.
Krishna
rimelek
January 31, 2022, 9:58pm
9
Oh… I see. I forgot about that part of your original message
I will bookmark this topic and when I have time to play with it, if I can find something out, I will come back. Until that if you really want it to be fixed, so you can use buildkit again, you should open an issue on GitHub or maybe just comment on the one I shared above.
