We currently use this to compress the docker save output:
docker save | xz -T 0 > image.tar.xz
With 29.5.3 the image.tar.xz size is 1.7 G, but with 29.6.0 the size is now 2.3 G, an increase of over .5 G - a very significant change in size!
It looks like 29.6.0 now compresses all of what were previously uncompressed tar file at a per file level.
That change seems to prevent better compression from happening at a higher level - that is, compressing across compressed files is not able to improve compression much as compared to compressing a bunch of uncompressed files.
Is there a way to get the behavior we had with 29.5.3 (and earlier), such that we can run compression across the uncompressed files?
Right now the only way I can think of to do this is to run the docker save, untar the result, uncompress compressed files and than re-tar and compress the result.
With the images I tested, I could not reproduce any difference between 29.6.1 and 29.5.3. I can’t see any relevant change in the release notes either.
If I test a much odler version, the exported layers are indeed different and the exported size is different. Some files are just json files some are targz files indeed. Not sure,because I never tested, but the change could also be caused by the new container image store. If that is the case, I guess if you don’t use that image store, that could help, but you can’t just disable it. You would need to pull all images again and optionally remove the images first from the container image store.
I was comparing our build server on Ubuntu 22.04 with docker 29.6.0 against a local “docker save” using 29.5.3 as I can’t see what our previous build server agent had - it was 28.x something - let alone easily run the older version or get a shell on the build server (VM that gets spun up).
I just tried again now, and can see the images are the same size on my local system when I try 29.5.3 vs 29.6.1 (went to 29.6.1 as I just updated it again).
But the image sizes are larger on our build system. So I can focus on what else is different on the build server.
Can you elaborate on what you mean by “the change could also be caused by the new container image store”?
If ypu chekck the shared link, the documentation explains what the containerd image store is. In short, the images are stored by containerd, not Docker directly there could be differences in the format.Since the image store can be changed, the problem could be unrelated to the version itself if the image store was different on the different machines. Containerd can also be configured separately, so while Docker can be configured to use containerd for storing images,
It is not about docker save, but you can find the Exporters documentation of docker buildx
My local system had been upgraded so was still using the overlay2 storage drive, if I use the newer overlayfs and driver-type: io.containerd.snapshotter.v1 it also uses compressed blobs in docker save output.
The build server is built from scratch (no upgrades) so it uses the overlayfs and driver-type: io.containerd.snapshotter.v1.
Short term we can probably use overlay2 but long term we want to use the newer storage driver, but need to get uncompressed blobs if we want to keep the docker save size reasonable (i.e. not use an extra 500 MB of space).
Any ideas how to use the newer storage driver and have no compression on the underlying files it includes in docker save?
Edit: I am still reading up on exporters and compression.
Indeed, starting with docker v29, fresh installations use the containerd snapshotter as image store by default. When existing pre v29 installation where updated, the configured storage driver remained untouched.
Probably irrelevant for you, but still noteworthy: older Docker versions (like sometimes found on NAS devices) might not support images that use zstd compression.
But that is a parameter for docker buildx build, so if that helps, that only helps with newly built images. I could not find anything to affect docker save directly either.
Temporarily, tools like skopeo can help to export images in zstd directly without using docker save. I tested it with ubuntu:latest