Images not building after upgrade to Docker for Mac 4.4.2 (73305)

[krishnaku]

Hello,

I have a two stage docker build process that creates a base image which has all the static dependencies of the app, and a second stage that used this base image to install the latest application code.

This process has been working fine for several years now, and started to fail after I upgraded from Docker 4.2.0 to Docker 4.4.2. I am not sure what changed in between these two releases, but searching on SO/Google etc did not turn up any relevant answers. so I am posting here to see if there might be some help available.

The error I am seeing is this:

#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed**

failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

But it is throwing this error when trying to look up the image metadata for the base image that It just finished building in the previous step. This image is local and not pushed to any registry, but it looks like the build process is trying to look this up from the docker.io registry.

NOTE: since I am a first time user, this forum seems to be rejecting my post because it has more than two links. So I am unable to post the actual logs which it thinks contains links,

---

Any help would be appreciated.

Krishna

[rimelek (Ákos Takács)]

@krishnaku

NOTE: since I am a first time user, this forum seems to be rejecting my post because it has more than two links. So I am unable to post the actual logs which it thinks contains links,

You shouldn’t share logs without using the </> button (code block or as the forum calls it “preformatted text”). Your logs could contain other characters too like * which would make a bullet list “italic” text.

example

this is the log with links : docker.io forums.docker.com

Those logs would be useful. Docker would not look for metadata online unless you built image got a different name than you expected or it wasn’t built at all in the previous step.

[krishnaku]

Thanks Akos… my bad for not figuring out the code block issue

I was able to get everything to build just by turning build_kit off. So that’s how I am working around the issue for now. So I suspect this is a bug of some sort for sure.

I’ll try and reproduce this with build_kit=1 and send you the logs.

[rimelek (Ákos Takács)]

@krishnaku

I was able to get everything to build just by turning build_kit off.

Build kit again… This is what I was talking about to @meyay in an other topic.

I know a little bit more about buildkit since then but I don’t know why it would cause this issue. When you enable buildkit, your images are built using runc directly. but it should not check metadata online of a local image either. When I am done with my docker build tutorial without buildkit, I should start investigating more how buildkit really works.

Your logs may help to understand your case if you can reproduce it.

[krishnaku]

Here is the complete build log.
This is log with the default setting (DOCKER_BUILDKIT=1)

Pulling registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Building package base image: polaris-utils-base with Builder Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and Release Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#1 [internal] load build definition from Dockerfile
#1 sha256:72a2373a5f034babd468efd358fd3fcb1e7ae60c6b70ddd9fbd89d59c4d6ce2a
#1 transferring dockerfile: 2.08kB done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:cf164168c4c6516cb0dd2d1d1b65c047c81ed6983a7ed0714f424c0725568fc0
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#3 sha256:d2bd26b30afc04ce4f9781ab6637055fcedd793ac63d48a928eb1af5834b793a
#3 ...

#4 [auth] polaris-common/polaris-build/polaris-build/polaris-python-build-base:pull token for registry.gitlab.com
#4 sha256:d8c2891d3d6648146009781d4dc867f1165be6d5eed53bef803fa59c47930f11
#4 DONE 0.0s

#3 [internal] load metadata for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
#3 sha256:d2bd26b30afc04ce4f9781ab6637055fcedd793ac63d48a928eb1af5834b793a
#3 DONE 1.9s

#5 [stage-0 1/7] FROM registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest@sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
#5 sha256:191f5d4a5139de326cb13ee047be13f1576c7c90908220920a46141e75864ec6
#5 DONE 0.0s

#6 [internal] load build context
#6 sha256:e1d5bf1134e2e0a47987ff78f0acbe5bed6b64d3fc726ed5132e67dc76bc85e4
#6 transferring context: 12.41kB 0.0s done
#6 DONE 0.0s

#13 [stage-1 2/4] COPY --from=0 /project/wheels /tmp/build/wheels/
#13 sha256:4c1857940cc31b91fc1d2bbe70e3bf1197d3ed2f9dd494f558852c1f2495feb3
#13 CACHED

#7 [stage-0 2/7] COPY  *requirements.txt       polaris-build/system-pip-constraints.txt       /tmp/build/
#7 sha256:a4d464168708cfda826cbeb8d0969a186baad22542e371ccfd10a6a1de00cf6c
#7 CACHED

#12 [stage-0 7/7] RUN pip freeze | grep -v polaris > /project/requirements.txt || true     && pip freeze | grep polaris > /project/polaris-requirements.txt || true
#12 sha256:8c5ce7233bc9b8ed77b4340f4a7e81650e399c091f7cbb67e2473b828ffe6d5c
#12 CACHED

#11 [stage-0 6/7] RUN mkdir -p /project/wheels &&     ./install.sh --dependencies-only
#11 sha256:d3e247cde9c8be2beb13b269cce5b1ec951118ae829d62ecbf1d019f889a22a8
#11 CACHED

#9 [stage-0 4/7] COPY . /project/
#9 sha256:2dfc6faba2b315b4a39c182883027d7bc62755cf4cfbb03a3197b8dfc3a77cc4
#9 CACHED

#10 [stage-0 5/7] WORKDIR /project
#10 sha256:fb6f3cf7d501af7f149e296d8cf45a8794bb64e19fbdd6794de567ce21edf541
#10 CACHED

#8 [stage-0 3/7] RUN  touch /tmp/build/all-requirements.txt      && cat /tmp/build/all-requirements.txt | grep -v polaris > /tmp/build/dependencies.txt || true      && pip install -r /tmp/build/dependencies.txt -c /tmp/build/system-pip-constraints.txt      && rm -rf /tmp/build
#8 sha256:5675b92fba82f8b91e405df3e0794fe133c0c3fcbbb2f94068c060489ebb156c
#8 CACHED

#14 [stage-1 3/4] COPY --from=0 /project/requirements.txt /project/polaris-requirements.txt /tmp/build/
#14 sha256:145586e0047f38264fe53bd733457fd985652b577f13a4aa605dd05c1c002409
#14 CACHED

#15 [stage-1 4/4] RUN cat /tmp/build/*requirements.txt | xargs -I '?' pip install --no-index --find-links /tmp/build/wheels ?     && rm -rf /tmp/build
#15 sha256:9f8a482e54d6ff8ffe14bd1f873cbb6d59fc77b2a4e5d29d9c351a97e4ad40a0
#15 CACHED

#16 exporting to image
#16 sha256:e8c613e07b0b7ff33893b694f7759a10d42e180f2b4dc349fb57dc6b71dcab00
#16 exporting layers done
#16 writing image sha256:2b416b81dab0fd8b5004f44495b7c4347c15cef0881b8dead5df2c4d357151f5 done
#16 naming to docker.io/library/polaris-utils-base done
#16 DONE 0.0s

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building package image: polaris-utils
#1 [internal] load build definition from Dockerfile
#1 sha256:ee306a68d614211594e97628b6c6b1fcf290338ee4e55ef2ad032bd355ec017e
#1 transferring dockerfile: 758B done
#1 DONE 0.0s

#2 [internal] load .dockerignore
#2 sha256:dfbfc751fd0d76f70a4f499f66c96d6ef2fa9b89d725abf07dc868e14dcd80e4
#2 transferring context: 2B done
#2 DONE 0.0s

#3 [internal] load metadata for docker.io/library/polaris-utils-base:latest
#3 sha256:4ac34a971b9e19a3ab5761a049fa8fa1f8c27e1cf07880f0fedb61795e3d972b
#3 ...

#4 [auth] library/polaris-utils-base:pull token for registry-1.docker.io
#4 sha256:37ec73dfef4fa6bbe5ba066bbe85ae6ef623005d3e51ae4f8ed869526e80224a
#4 DONE 0.0s

#3 [internal] load metadata for docker.io/library/polaris-utils-base:latest
#3 sha256:4ac34a971b9e19a3ab5761a049fa8fa1f8c27e1cf07880f0fedb61795e3d972b
#3 ERROR: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
------
 > [internal] load metadata for docker.io/library/polaris-utils-base:latest:
------
failed to solve: rpc error: code = Unknown desc = failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Error: build package failed.

Here is the log with DOCKER_BUILDKIT=0

Krishnas-Mac-Mini-M1:polaris-utils krishna$ DOCKER_BUILDKIT=0 package build
Pulling registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
latest: Pulling from polaris-common/polaris-build/polaris-build/polaris-python-build-base
Digest: sha256:9ada2f789852c91b9e89aad039ae58c7da23069629991f96f4545523196046cc
Status: Image is up to date for registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Building package base image: polaris-utils-base with Builder Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest and Release Image: registry.gitlab.com/polaris-common/polaris-build/polaris-build/polaris-python-build-base:latest
Sending build context to Docker daemon  519.9kB
Step 1/13 : ARG BUILDER_IMAGE
Step 2/13 : ARG RELEASE_IMAGE
Step 3/13 : FROM ${BUILDER_IMAGE}
 ---> c14a1fddf37c
Step 4/13 : COPY  *requirements.txt       polaris-build/system-pip-constraints.txt       /tmp/build/
 ---> Using cache
 ---> 4700e7a74924
Step 5/13 : RUN  touch /tmp/build/all-requirements.txt      && cat /tmp/build/all-requirements.txt | grep -v polaris > /tmp/build/dependencies.txt || true      && pip install -r /tmp/build/dependencies.txt -c /tmp/build/system-pip-constraints.txt      && rm -rf /tmp/build
 ---> Using cache
 ---> 34f8729066ea
Step 6/13 : COPY . /project/
 ---> Using cache
 ---> fba3d292f3eb
Step 7/13 : WORKDIR /project
 ---> Using cache
 ---> 9c21390dc85e
Step 8/13 : RUN mkdir -p /project/wheels &&     ./install.sh --dependencies-only
 ---> Using cache
 ---> 446df7f62b85
Step 9/13 : RUN pip freeze | grep -v polaris > /project/requirements.txt || true     && pip freeze | grep polaris > /project/polaris-requirements.txt || true
 ---> Using cache
 ---> 72be2b1f6872
Step 10/13 : FROM ${RELEASE_IMAGE}
 ---> c14a1fddf37c
Step 11/13 : COPY --from=0 /project/wheels /tmp/build/wheels/
 ---> Using cache
 ---> a6db9b561cca
Step 12/13 : COPY --from=0 /project/requirements.txt /project/polaris-requirements.txt /tmp/build/
 ---> Using cache
 ---> 032adeadb813
Step 13/13 : RUN cat /tmp/build/*requirements.txt | xargs -I '?' pip install --no-index --find-links /tmp/build/wheels ?     && rm -rf /tmp/build
 ---> Using cache
 ---> f789f46bc7ac
Successfully built f789f46bc7ac
Successfully tagged polaris-utils-base:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
Building package image: polaris-utils
Sending build context to Docker daemon  519.9kB
Step 1/12 : ARG PACKAGE_BASE_IMAGE
Step 2/12 : FROM ${PACKAGE_BASE_IMAGE}
 ---> f789f46bc7ac
Step 3/12 : ARG ALEMBIC_ROOT
 ---> Using cache
 ---> 5f22825b2355
Step 4/12 : ENV ALEMBIC_ROOT ${ALEMBIC_ROOT:-.}
 ---> Using cache
 ---> 10bbe751ad8a
Step 5/12 : COPY . /project/
 ---> Using cache
 ---> 050fc5cde3fa
Step 6/12 : WORKDIR /project
 ---> Using cache
 ---> 657b59bbce96
Step 7/12 : RUN ./install.sh     && rm -rf /project/*
 ---> Using cache
 ---> 818ff9090dbd
Step 8/12 : COPY conftest.py pytest.ini /project/
 ---> Using cache
 ---> 704fbc684559
Step 9/12 : COPY test /project/test/
 ---> Using cache
 ---> 990181fc84a8
Step 10/12 : COPY polaris-build/bin/* bin/* /usr/local/bin/
 ---> Using cache
 ---> 69dc18c78067
Step 11/12 : COPY polaris-build/ignore-missing ${ALEMBIC_ROOT}/alembic.ini*  /package_migrations/
 ---> Using cache
 ---> b8ec45a81776
Step 12/12 : COPY polaris-build/ignore-missing ${ALEMBIC_ROOT}/migrations*  /package_migrations/migrations/
 ---> Using cache
 ---> 2f027be3d6ae
Successfully built 2f027be3d6ae
Successfully tagged polaris-utils:latest

Use 'docker scan' to run Snyk tests against images to find vulnerabilities and learn how to fix them
 Executing post build steps for python package..
WARNING: The requested image's platform (linux/amd64) does not match the detected host platform (linux/arm64/v8) and no specific platform was requested
Please note: all-requirements.txt has been updated. Please commit this change.

[krishnaku]

Also I can confirm that this is not a 4.4.2 issue. I was able to reproduce this with 4.2.0 as well. Another bit of perhaps important info: this build is running on a new Mac with the M1 chip, whereas all my other machines are running the intel chip.

[rimelek (Ákos Takács)]

Thank you for the logs! Sorry I am tring to catch up with the unread posts.

@krishnaku

this build is running on a new Mac with the M1 chip

After I read only your previous post and searched for the error message, I would have asked if you had a Mac with the M1 chip, because it looks like it was not really compatible with Docker Compose, especually with buildkit.

Failed to solve with frontend dockerfile.v0: failed to create LLB definition: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

I tried to try your image , but I actualy can’t pull it, doesn’t matter where I try (Mac M1 or Linux AMD64).
I can’t find any image on Docker Hub with this name. Is it some kind of special image that only you have access to or is this a part of the bug that Docker sees it as an official image?

[krishnaku]

I am using Docker Compose fine on this machine (my entire dev stack is built and run using this) with buildit turned off. My point is that the image is never published to Docker Hub. I actually don’t use docker hub at all - my registries are on GitLab. But in this case, there is no real registry involved anywhere, I am creating a local image in one step of the build and using that local image in the next step of the build process and for some reason it seems to be looking on Docker Hub for that intermediate image.

Anyway, I hope this can get fixed, in the meantime, I have moved all my actual production builds to a Linux machine and so I am not blocked by this anymore. But it still seems like an important bug to fix for folks who are using Docker Compose and newer Macs to build images.

Krishna

[rimelek (Ákos Takács)]

@krishnaku

But in this case, there is no real registry involved anywhere

Oh… I see. I forgot about that part of your original message

I will bookmark this topic and when I have time to play with it, if I can find something out, I will come back. Until that if you really want it to be fixed, so you can use buildkit again, you should open an issue on GitHub or maybe just comment on the one I shared above.