Issues with remote cache on Docker Compose

Docker Engine Compose
1

Hi,

I saw some discussions about my issue, but some are quite old/outdated (4-5 years ago).
I have a complex docker build system, and I’m using docker compose for build orchestration (using depends on) and parallelism (building few images which have common dependencies in parallel).I’m struggling with remote cache using docker compose, it just won’t work (cache is invalidated) when building more complex layers. It does work when experimenting in very simple layers for testing purpose (e.g. touch /some_file for the matter of the test).
I’m trying to build an image on one builder machine, export it to remote cache of repository type, and then expect a cache hit on another builder machine, without the need to rebuild the image. It just won’t work, the last layer is invalidated.
As of now, after reading multiple places discussions, I make sure that:

  1. I’m exporting and also passing as build arg:
    • export DOCKER_BUILDKIT=1
    • export COMPOSE_DOCKER_CLI_BUILD=1
    • export BUILDKIT_INLINE_CACHE=1
  2. I’m using the exact docker, docker compose and Buildkit plugin versions:
    • Docker version 25.0.4, build 1a576c5
    • Docker Compose version v2.29.2
    • Docker buildx/buildkit version: GitHub - docker/buildx: Docker CLI plugin for extended build capabilities with BuildKit v0.16.2 99dea6dacacc3d604788953088560b9880550570
  3. I also make sure I build from the exact Git commit hash, and that my working environments (context) are clean:
    • git reset –hard
    • git clean -fdx
  4. Comparing the images using Google’s container-diff tool (GitHub - GoogleContainerTools/container-diff: container-diff: Diff your Docker containers) gives no results, images are identical. Here is the command:
    • container-diff diff --type=history --type=apt --type=node --type=pip --type=rpm --type=size --type=file --type=aptlayer --type=emerge --type=layer --type=metadata --type=rpmlayer --type=sizelayer

Here Is how I define the caching in my docker compose:

x-build-cache-setting: &build-cache-setting
cache_from:
- type=registry,ref=${CACHE_URL}
cache_to:
- type=registry,ref=${CACHE_URL},${CACHE_TO_OPTIONS}

While:
• CACHE_URL is some-url:/image-name:latest
• CACHE_TO_OPTIONS=mode=max,ignore-error=true,compression=zstd,compression-level=3,image-manifest=true

Trying to build with direct docker commands seems to be working, by the way.
Anything I’m doing wrong?

2

Compose is a client If you share that command we can check if we see any difference between what the command and compose would do. You can also use

docker compose config

to see how the yaml file was interpreted.