Docker push: Force Push All Layers

[davidbernat]

I notice during buildx multi-architecture rebuilds that the ‘docker push’ command often does not update layers, even when changes are made. Furthermore, a manual override to ensure all layers in the repository are fresh would be a boon to debugging. I recommend a --force-all-layers flag be added to ‘docker push.’ If such a flag already exists, then I do not know about this flag or this flag is not documented in the ‘docker push’ documentation. Thank you!

[rimelek (Ákos Takács)]

You can’t push a multiarch image with docker push. After running docker buildx build --platform linux/amd64,linux/arm64 you will have the arm64 and amd64 images in the local build cache unless you use the --push option as well:

docker buildx build --platform linux/amd64,linux/arm64 . -t dockerhubusername/imagename --push

I guess you had an image with the same name built with docker build using only one architecture and you pushed that image. If you do that it also means you will loose the multiarch image on Docker Hub and you will have only that one architecture you pushed the last time.

[davidbernat]

To completely summarize what you are saying here so that we are on the same page:

1. buildx --platform platform1,platform2 builds multi-architecture docker containers, and is the only way to build a multi-architecture build.
2. However, docker push does not possess the capability to push the multi-architecture containers. If you push the multiarch containers using docker push, one or more of the architectures will be stripped out; only one architecture will be pushed.
3. The only way to push a multi-architecture build is to use the --push flag at the time of using docker build buildx ....

This is a very helpful clarification.
A. I do not believe this is stated in the documentation.
B. What is the design decision behind that choice, as pushing concurrently with building prevents the user from testing the docker container locally with docker run before making the slow, bandwidth intensive push, hence impacting iteration cycles.

In my particular case, I know I debugged by pushing with both methods, so this is unlikely to solve my specific problem. But I will make a check tonight to confirm that. I also deleted all my hub versions to ensure a fresh, fully upload.

C. There are a few edge cases I wonder about: for instance, if I have a single architecture docker in the hub, and push using docker build buildx ... --push will the multi-architecture build arrive in the hub, or will the multi-architecture be stripped down to the single-architecture?

I ask, in part, because I prefer the version history of my containers, and want to keep a mass delete from the repository to a minimum. It also helps to be as precise as possible when executing code across architectures in the cloud.

Thank you very much. You are clearly very knowledgable and I appreciate you sharing this with the forum.

[rimelek (Ákos Takács)]

@davidbernat

buildx --platform platform1,platform2 builds multi-architecture docker containers, and is the only way to build a multi-architecture build.

It builds images, not containers and no, this is not the only way, but if you are building with buildx then this is the way. You could also use docker manifest to combine multiple images with different architectures into one multiarch image (experimental feature), but I only used buildx.

@davidbernat

If you push the multiarch containers using docker push, one or more of the architectures will be stripped out; only one architecture will be pushed.

Yes. docker push was invented before multiarch images and you can’t have a multiarch image locally. Everything that you see in the output of docker image ls is a single image having only one architecture. These are the images (again, not containers) that you can push with docker push.

@davidbernat

The only way to push a multi-architecture build is to use the --push flag at the time of using docker build buildx ...

No, as it is mentioned above.

@davidbernat

A. I do not believe this is stated in the documentation.

Maybe not directly. buildx helps you to run the build command on one machine which runs the actual build on different machines that supports an architetcure. Since we also has QEMU which can emulate different architectures, you can create a builder locally as a container which will emulate each supported architecture for each build: docker buildx create --name multiarch

@davidbernat

B. What is the design decision behind that choice, as pushing concurrently with building prevents the user from testing the docker container locally with docker run before making the slow, bandwidth intensive push, hence impacting iteration cycles.

After you ran the build using buildx without the --push flag, you can load the different architectures into different images like:

docker buildx build . \
  ---platform linux/arm64 -t localhost/imagename:arm64 --load
docker buildx build . \
  ---platform linux/amd64 -t localhost/imagename:amd64 --load

test them and run the buildx build with the --push flag. This is something I found out reading about the flags and playing with Docker. I don’t know if it is mentioned anywhere, but I guess there are some blog posts and videos about it. I will check it and if I don’t find a lot of content, I will create one. Thanks for the idea

@davidbernat

C. There are a few edge cases I wonder about: for instance, if I have a single architecture docker in the hub, and push using docker build buildx ... --push will the multi-architecture build arrive in the hub, or will the multi-architecture be stripped down to the single-architecture?

Every push overwrites the image tag in the registry. The image itself will not disappear immediately, but the tag will be removed from it. If you know tha ID of the old version, you can still pull it and retag it. If you use tags properly, you will never need to worry about it, since every one of your builds will have different tags so you will not overwite anything. Maybe only the latest tag if you have one or other aliases like v2.1 pointing to v2.1.2 or v2.1.3. You can also add build numbers. CI/CD providers generate it by default ad a variable.

[davidbernat]

A few notes and responses to continue the conversation and for future readers:

1. “you can’t have a multiarch image locally” – this is very useful to know. I was under the impression that the image itself was multi-architecture; but what you are saying is that by using buildx with --push an entity is created in the docker repository which is capable of generating one of multi-architectures; then, when you pull the image, the architecture-specific version you specify is pulled.

2. “you can create a builder locally as a container which will emulate” – the container created with docker buildx create locally has the capacity to generate an architecture-specific images, because of QEMU, which can emulate the specific architecture necessary.

3. " After you ran the build using buildx without the --push flag, you can load the different architectures into different image" Got it. But, you can not push the “multi-architecture capable” container without having used the --push flag in docker build buildx.

This all feels unintuitive because of the design choices in the API, not the requirements of operating multi-architecture capable containers and architecture-specific images. An official document may be useful.