Docker image save troubles with overlayFS, but works with overlay2?

Docker Engine General
1

On 2 computers, a laptop and a desktop, both running Ubuntu 24.04.4 LTS, with docker engine

$ docker --version
Docker version 29.4.3, build 055a478
$ docker info
...
 Storage Driver: overlayfs
  driver-type: io.containerd.snapshotter.v1
...
$ containerd --version
containerd containerd.io v2.2.3 77c84241c7cbdd9b4eca2591793e3d4f4317c590

every time I do a docker image save -o outfile.tar someimage
sometimes I get a much smaller (and incomplete) .tar file that (for sure) does not contain all layers of the image.
(Some images are saved properly, others are not. Did not figure out what works and what does not work, but the behavior is consistent across my 2 computers).

The only way I could work around the problem (after lots of unsuccessful web search and frustrated chatbot tipsā€¦) was to let docker re-create the data-root after downgrading the storage driver from the default overlayfs to overlay2. ( https://docs.docker.com/engine/storage/drivers/select-storage-driver/ )

Any idea of what is going on ? Am I the only one ?

Best regards,

PS: This is my first post here, because I did not find any similar search results on Google or ChatGPT.
If I violated any rule, sorry in advance, and I would appreciate pointing me in the right direction.

2

No, you did great.

Can you share some images that worked and some that didnā€™t? Or were those private images?

3

So far, I could not reproduce it with images directly pulled from docker hub. I am trying to look for an open-source exampleā€¦

The issue might be related to images that come from a gitlab container registry: GitLab container registry | GitLab Docs

4

Do you see any pattern in the base images? For example images from the same base image work, others donā€™t?

5

All the images that had problems were built (docker build ... -f Dockerfile ...) on an older

$ docker info
Client: Docker Engine - Community
 Version:    28.1.1
...
 Server Version: 28.1.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false

and then pushed on to a gitlab container registry.

6

The main difference is that new installation of Docker v29 use the contained image store, while old installation use the built-in storage driver of Docker.

Both will use different media types:

docker image inspect --format='{{.Descriptor.mediaType}}' <image>

The migration towards the containerd snapshotter is in preparation for future features that will rely on it. So sticking to the old storage driver is not a long term solution.

I remember we had a discussion about it, let me check if the forum search helps finding the topic.

7

There are a couple of issues regarding docker save open in the Moby GitHub project

Are the images that fail multi-arch images?

Have you tried if it makes a difference if you include the base image(s) in the command as well? It could be related to the issue that base image layers are ignored when saving an image created from it:

8

(For an image - letā€™s call it original-image-tag:1.2.3 that had an incomplete docker save) Iā€™ve tryed to include the base image, like for example node:24.14.1-trixie-slim:

docker image save -o my-full-image.tar orginal-image-tag:1.2.3 node:24.14.1-trixie-slim

The tar file was much bigger.
To test, Iā€™ve deleted docker rmi ... all related images from the laptop (including orginal-image-tag:1.2.3 node:24.14.1-trixie-slim), and loaded the saved .tar image

$ docker image load -i ./my-full-image.tar
Loaded image: orginal-image-tag:1.2.3
Loaded image: node:24.14.1-trixie-slim

and the image executed without issues.

So, I also believe it is the same problem. At least now I know how to work around the issue (until one day I can upgrade the building of the image).

Thank you all for the help (and you can close this issue).