Cannot pull latest image on Raspberry Pi 4

General Discussions
1
  • Platform information:
    • Hardware: Raspberry Pi 4
    • OS: Raspberry Pi OS / x64 / 11 (bullseye)
    • Docker version 20.10.23, build 7155243

Hey :wave:t2:

I’ve been using the image mu88/shopandeat:latest for years on my Raspberry Pi 4. One month ago, I switched to ASP NET Core 7.0 and everything worked smoothly.

Today I made a little fix in the app and created a new version of mu88/shopandeat:latest. When trying to pull this image on my Raspi via docker pull mu88/shopandeat:latest, this fails with:

latest: Pulling from mu88/shopandeat
no matching manifest for linux/arm64/v8 in the manifest list entries

Which is strange because when logging into Docker Hub it seems to be okay:

grafik
grafik798×177 22.5 KB

How can I resolve this issue?

Thx and regards!

2

This is a friendly push

3

I was able to pull the image using the digest:

docker pull mu88/shopandeat@sha256:50ecba35f259e2d40f59301cdc07e766dac45d86f3906615fdfb9a6de952a0ee

but the latest tag doesn’t work.

It could be because the latest tag is actually not “latest” just veery similar and we can’t tell the difference, but pull doesn’t work even when I click on the copy button of the tag. Your image is indeed linux/arm64/v8, but for some reason the tag doesn’t work. Can you try to push it again using a different tag and also “latest”?

If we can’t fix the tag, only the Docker support could help or you could also report the issue on GitHub in the hub-feedback repository.

4

Thank you for your help! I’ve triggered another run of my GitHub Action and published latest and v2.4 - both cannot be pulled on my Raspi. However, I can pull both tags on my Win11 dev notebook. What does that mean?

5

Docker Desktop for Windows can emulate architectures. If your Windows machine has amd64 CPU and you could pull arm images, that could mean that the tag actually exists, but the detection of the architecture doesn’t work for some reason or it is not an arm64 image and it just looks like it.

How do you build the image? Can you share the command?
And alos where do you build it?

I still can’t pull your image even though my macOS has the same CPU architecture that I can see on Docker Hub

6

Very interested, thank you for the explanation.

I’m building the image via this GitHub Action: ShopAndEat/Release.yml at main · mu88/ShopAndEat · GitHub

And I’m doing so for quite some time, until now without any issues.

7

I think we found a bug in Docker Hub. I can’t pull my images either: https://hub.docker.com/r/itsziget/httpd24
I have built amd64 and arm64 images too. I actually built linux/arm64/v8 but for some reason it became linux/arm64 but still worked on my macOS, but not anymore.

update:

Or the bug could be in Docker, since I could pull the image in a virtual machine where I had older Docker version. I am updating my system to check with the newer Docker version.

update 2:

I was wrong… I can pull my image on Mac and in the Linux VM too. I must have used a wrong name before. But I can’t pull yours.

8

Try this on Windows 11

hub-tool tag ls mu88/shopandeat --format json

It will show this:

[
  {
    "Name": "mu88/shopandeat:latest",
    "FullSize": 120557472,
    "LastUpdated": "2023-01-25T16:35:01.717802Z",
    "LastUpdaterUserName": "mu88",
    "Images": [
      {
        "Digest": "sha256:e27e405ce56030572339b74b4a1a23a39d5f648f45b043ed0d24cc34c06164e0",
        "Architecture": "arm64",
        "Os": "linux",
        "Variant": "v8",
        "Size": 120557472,
        "LastPulled": "2023-01-25T16:35:02.422923Z",
        "LastPushed": "2023-01-25T16:35:01.292494Z",
        "Status": "active"
      },
      {
        "Digest": "sha256:74a8ceae702f912cf38ecbab4a6f3220cadcc33a23199bc2e1cdb20ea2e85be8",
        "Architecture": "unknown",
        "Os": "unknown",
        "Variant": "",
        "Size": 21633,
        "LastPulled": "0001-01-01T00:00:00Z",
        "LastPushed": "2023-01-25T16:35:01.005212Z",
        "Status": "active"
      }
    ],
    "LastPulled": "2023-01-25T16:36:50.283509Z",
    "LastPushed": "2023-01-25T16:35:01.717802Z",
    "Status": "active"
  },
  {
    "Name": "mu88/shopandeat:v2.4",
    "FullSize": 120557472,
    "LastUpdated": "2023-01-25T16:35:03.084345Z",
    "LastUpdaterUserName": "mu88",
    "Images": [
      {
        "Digest": "sha256:e27e405ce56030572339b74b4a1a23a39d5f648f45b043ed0d24cc34c06164e0",
        "Architecture": "arm64",
        "Os": "linux",
        "Variant": "v8",
        "Size": 120557472,
        "LastPulled": "2023-01-25T16:35:02.422923Z",
        "LastPushed": "2023-01-25T16:35:01.292494Z",
        "Status": "active"
      },
      {
        "Digest": "sha256:74a8ceae702f912cf38ecbab4a6f3220cadcc33a23199bc2e1cdb20ea2e85be8",
        "Architecture": "unknown",
        "Os": "unknown",
        "Variant": "",
        "Size": 21633,
        "LastPulled": "0001-01-01T00:00:00Z",
        "LastPushed": "2023-01-25T16:35:01.005212Z",
        "Status": "active"
      }
    ],
    "LastPulled": "2023-01-25T16:36:50.283509Z",
    "LastPushed": "2023-01-25T16:35:03.084345Z",
    "Status": "active"
  },
  {
    "Name": "mu88/shopandeat:5.0-buster-slim-arm32v7",
    "FullSize": 94906365,
    "LastUpdated": "2022-01-07T13:18:27.035796Z",
    "LastUpdaterUserName": "mu88",
    "Images": [
      {
        "Digest": "sha256:7e426540375daba3cbf097c7db243b029bef7b680cce4b82ee4ff94d425a285a",
        "Architecture": "arm",
        "Os": "linux",
        "Variant": "",
        "Size": 94906365,
        "LastPulled": "2022-01-07T13:18:27.243202Z",
        "LastPushed": "2022-01-07T13:18:27.035796Z",
        "Status": "inactive"
      }
    ],
    "LastPulled": "2022-01-07T13:18:27.243202Z",
    "LastPushed": "2022-01-07T13:18:27.035796Z",
    "Status": "inactive"
  }
]

Somehow an “unknown” architecture was pushed after arm64.

      {
        "Digest": "sha256:74a8ceae702f912cf38ecbab4a6f3220cadcc33a23199bc2e1cdb20ea2e85be8",
        "Architecture": "unknown",
        "Os": "unknown",
        "Variant": "",
        "Size": 21633,
        "LastPulled": "0001-01-01T00:00:00Z",
        "LastPushed": "2023-01-25T16:35:01.005212Z",
        "Status": "active"
      }

It shouldn’t affect docker pull since there is an arm64 image too, but this could be the reason why Docker is confused. Try to push an arm image manually.

docker pull mu88/shopandeat@sha256:7e426540375daba3cbf097c7db243b029bef7b680cce4b82ee4ff94d425a285a
docker tag  mu88/shopandeat@sha256:7e426540375daba3cbf097c7db243b029bef7b680cce4b82ee4ff94d425a285a mu88/shopandeat:v2.4-test

Then try to pull mu88/shopandeat:v2.4-test on RPi4.

I pushed a copy of your image to one of my test repos temporarily. You can try that too.

docker pull rimelek/test2

As you can see below Docker Hub recognized your image and mounted the layers from that.

docker push rimelek/test2

Using default tag: latest
The push refers to repository [docker.io/rimelek/test2]
09aea82818d2: Mounted from mu88/shopandeat
5f70bf18a086: Mounted from mu88/shopandeat
7473d26c60b8: Mounted from mu88/shopandeat
6428507b6b5f: Mounted from mu88/shopandeat
373d1d3dc011: Mounted from mu88/shopandeat
f4dc846ec08e: Mounted from mu88/shopandeat
827ee43f735d: Mounted from mu88/shopandeat
1c14ce4d3e6d: Mounted from mu88/shopandeat
latest: digest: sha256:7e426540375daba3cbf097c7db243b029bef7b680cce4b82ee4ff94d425a285a size: 1996
1 Like
9

You’re absolutely right :muscle:t2:with your approach I can pull the image again. Thank you so much so far!

How do we continue?

10

How do you set the CPU architecture (platform) for github actions? The unknown platform could be caused by a wrong list of platforms, a bug in GitHub Actions or a bug In Docker Hub. Or more than one of these at the same time. I don’t know why Docker Hub would allow you to upload manifests with “unknown” architecture, but I don’t know why GitHub Actions would generate that. I built my httpd image on CircleCI using buildx and I could push the images correctly. The differnce is that I defined the platform for the docker buildx build command using a multiarch base image. I don’t see the platform definition in your yaml and the tag of your base image contains arm64 so it must be a single arch image. Docker could detect that you have an arm64 base image and generate an “unknown” architecture because you don’t have “platform” defined anywhere (at least I don’t know where else you can define it) and also generate an arm64 because of the base image.

This is just a big guess, but if it is what happens, defining the platform option in the yaml could solve the issue. You could first try to write a docker build command locally using buildx that builds and pushes the same image that GitHub actions does. Ifit works locally, the error could be in GitHub Actions. If not, then it could be a bug of Docker Hub or the buildx Docker cli plugin itself.

You probably know this page, but I leave this link here for anyone else
https://github.com/marketplace/actions/docker-setup-buildx#customizing

11

Okay, it’s getting ridiculous :sweat_smile: when setting the platforms parameter to either linux/arm64 or linux/arm64/v8 (see here), the build fails with cryptic errors in the .NET compilation part of my Dockerfile. Unfortunately I’m not allowed to upload the build log, so I’m just sharing some lines from the build log:

#15 [build  9/12] RUN dotnet restore "ShopAndEat/ShopAndEat.csproj"
#15 13.22   Determining projects to restore...
#15 20.77 /usr/share/dotnet/sdk/7.0.102/NuGet.targets(1179,7): error MSB4018: The "MSBuild" task failed unexpectedly. [/src/DataLayer/DataLayer.csproj]
#15 20.77 /usr/share/dotnet/sdk/7.0.102/NuGet.targets(1179,7): error MSB4018: System.NullReferenceException: Object reference not set to an instance of an object. [/src/DataLayer/DataLayer.csproj]
#15 20.77 /usr/share/dotnet/sdk/7.0.102/NuGet.targets(1179,7): error MSB4018:    at InvokeStub_MSBuild.get_TargetOutputs(Object, Object, IntPtr*) [/src/DataLayer/DataLayer.csproj]
#15 20.77 /usr/share/dotnet/sdk/7.0.102/NuGet.targets(1179,7): error MSB4018:    at System.Reflection.MethodInvoker.Invoke(Object obj, IntPtr* args, BindingFlags invokeAttr) [/src/DataLayer/DataLayer.csproj]

As soon as I remove the platforms parameter, the build works without any issues.

But I stumbled across the following lines in the build log:

2023-01-27T15:46:59.5627913Z ##[group]Installing QEMU static binaries
2023-01-27T15:46:59.5628652Z [command]/usr/bin/docker run --rm --privileged tonistiigi/binfmt:latest --install all
2023-01-27T15:46:59.8708768Z installing: s390x OK
2023-01-27T15:46:59.8709262Z installing: riscv64 OK
2023-01-27T15:46:59.8709617Z installing: mips64le OK
2023-01-27T15:46:59.8709952Z installing: mips64 OK
2023-01-27T15:46:59.8710281Z installing: arm64 OK
2023-01-27T15:46:59.8710922Z installing: arm OK
2023-01-27T15:46:59.8711744Z installing: ppc64le OK
2023-01-27T15:46:59.8956529Z {
2023-01-27T15:46:59.8956821Z   "supported": [
2023-01-27T15:46:59.8957266Z     "linux/amd64",
2023-01-27T15:46:59.8957491Z     "linux/arm64",
2023-01-27T15:46:59.8957717Z     "linux/riscv64",
2023-01-27T15:46:59.8957948Z     "linux/ppc64le",
2023-01-27T15:46:59.8958157Z     "linux/s390x",
2023-01-27T15:46:59.8958368Z     "linux/386",
2023-01-27T15:46:59.8958590Z     "linux/mips64le",
2023-01-27T15:46:59.8958818Z     "linux/mips64",
2023-01-27T15:46:59.8959039Z     "linux/arm/v7",
2023-01-27T15:46:59.8959257Z     "linux/arm/v6"
2023-01-27T15:46:59.8959461Z   ],
2023-01-27T15:46:59.8959645Z   "emulators": [
2023-01-27T15:46:59.8959856Z     <<left out for brevity>>
2023-01-27T15:46:59.8963179Z   ]
2023-01-27T15:46:59.8963362Z }
2023-01-27T15:47:00.0118621Z ##[endgroup]

Why does the supported array not contain linux/arm64/v8?

12

According to this wiki

ARMv8 started to support 64 bits. I guess arm64/v8 is just an alias to arm64, but I was surprised too when I experienced what I mentioned in one of my previous comments

Building multiplatform images usually requires emulation. Emulation is not perfect which could be one reason why you get error messages and I have seen multiple failing dotNET builds on the forum, but this is the point where I am out of ideas since you mentioned the build worked before and you could actually push an image that looks like an arm64 image. To solve this you need someone who understands dotNET error messages and builds better than I do.

13

Okay, very strange… :thinking:

So what would you recommend where to start: asking in the repo of the GitHub Action? Or in one Microsoft’s official .NET repos?

14

First I would recommend to ask someone who understands what this error message means. So asking someone in a Microsoft forum can be a good idea if you don’t have a personal friend who can help you.

For me the error message looks like a programming issue, but it is possible that it could be caused by building on a not supported architecture or incompatible parameters. I am not sure.

15

Okay thank you so much. Have a nice weekend!

16

I’ve opened this issue in Microsoft’s GitHub repo

17

If I am not mistaken, that is the repository of the base docker image. I suggested to ask on a Microsoft forum, but a GitHub repository would be good too if that repository is related to GitHub CI, not your base image. Sorry for my confusing previous reply.

I am not sure if that is the best place, but I found a bug report page of GitHub

Or you can also try the Microsoft Q&A

If you find out it must be a Docker Hub issue then Hub Feedback is what you will need

18

Finally I resolved this issue without any changes to the Dockerfile or my GitHub Actions, only by deleting all Docker tags of mu88/shopandeat. So mu88/shopandeat:latest somehow became corrupt, don’t know why :man_shrugging:t2:

Thank you once again @rimelek for spending your precious time!

1 Like