I have a CI process (GitHub Actions) that builds a large image several times. The image should not be changing except in the last few layers, so all image builds after the first should be very quick. However, on the CI machines, there is a very expensive RUN command that Docker is not using the cached layer for, which takes a typically < 15m CI run to over 40m. (Running the tests locally on my Mac, the cache is used as expected.)
From the logs and inspecting the image history, I can see that all layers before the RUN command are cached and the cached versions are used in subsequent builds; the expensive RUN command is the first layer that is not used from the cache. The command does not change between builds.
From the layer caching docs, my understanding was that RUN layers are only re-executed if a previously layer is invalidated or if the text of the command changed. Neither of these is the case; the RUN command is the first layer whose cache is invalidated and the text of the command does not change.
Here is the full CI run. Relevant portions picked out below.
Dockerfile:
ARG BASE_IMAGE=ubuntu:22.04
FROM ${BASE_IMAGE}
ARG DEBIAN_FRONTEND=noninteractive
RUN apt-get update && \
apt-get install -y curl unzip dos2unix wget && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /var/tmp/*
RUN mkdir -p /autograder/source && \
mkdir -p /autograder/submission && \
mkdir -p /autograder/results
ADD run_autograder /autograder/run_autograder
ADD setup.sh environment.yml requirements.* /autograder/source/
RUN dos2unix /autograder/run_autograder /autograder/source/setup.sh && \
chmod +x /autograder/run_autograder && \
apt-get update && bash /autograder/source/setup.sh && \
apt-get clean && \
rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
# create an empty submission_metadata.json file so that plugins don't error when trying to read it
RUN touch /autograder/submission_metadata.json
ADD otter_config.json run_otter.py /autograder/source/
ADD files* /autograder/source/files/
ADD tests /autograder/source/tests/
ENV PATH=/root/miniforge3/bin:$PATH
ADD __otter-grader /tmp/otter-grader
RUN mamba run -n otter-env pip install /tmp/otter-grader
RUN rm -rf /tmp/*
docker image history --no-trunc otter-grade:otter-grade-test after 1st build:
IMAGE CREATED CREATED BY SIZE COMMENT
sha256:747f867330c476819d139f2848eefbdb3a7aea35211c25aaa012e5b161c0b30c 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c rm -rf /tmp/* # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c mamba run -n otter-env pip install /tmp/otter-grader # buildkit 1.43MB buildkit.dockerfile.v0
<missing> 4 minutes ago ADD __otter-grader /tmp/otter-grader # buildkit 68.9MB buildkit.dockerfile.v0
<missing> 4 minutes ago ENV PATH=/root/miniforge3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0B buildkit.dockerfile.v0
<missing> 4 minutes ago ADD tests /autograder/source/tests/ # buildkit 3.64kB buildkit.dockerfile.v0
<missing> 4 minutes ago ADD files* /autograder/source/files/ # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago ADD otter_config.json run_otter.py /autograder/source/ # buildkit 215B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c touch /autograder/submission_metadata.json # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c dos2unix /autograder/run_autograder /autograder/source/setup.sh && chmod +x /autograder/run_autograder && apt-get update && bash /autograder/source/setup.sh && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # buildkit 5.48GB buildkit.dockerfile.v0
<missing> 8 minutes ago ADD setup.sh environment.yml requirements.* /autograder/source/ # buildkit 1.46kB buildkit.dockerfile.v0
<missing> 8 minutes ago ADD run_autograder /autograder/run_autograder # buildkit 219B buildkit.dockerfile.v0
<missing> 8 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c mkdir -p /autograder/source && mkdir -p /autograder/submission && mkdir -p /autograder/results # buildkit 0B buildkit.dockerfile.v0
<missing> 8 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt-get update && apt-get install -y curl unzip dos2unix wget && apt-get clean && rm -rf /var/lib/apt/lists/* /var/tmp/* # buildkit 8MB buildkit.dockerfile.v0
<missing> 8 minutes ago ARG DEBIAN_FRONTEND=noninteractive 0B buildkit.dockerfile.v0
<missing> 3 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 3 months ago /bin/sh -c #(nop) ADD file:ebe009f86035c175ba244badd298a2582914415cf62783d510eab3a311a5d4e1 in / 77.9MB
<missing> 3 months ago /bin/sh -c #(nop) LABEL org.opencontainers.image.version=22.04 0B
<missing> 3 months ago /bin/sh -c #(nop) LABEL org.opencontainers.image.ref.name=ubuntu 0B
<missing> 3 months ago /bin/sh -c #(nop) ARG LAUNCHPAD_BUILD_ARCH 0B
<missing> 3 months ago /bin/sh -c #(nop) ARG RELEASE 0B
image history after second build; note that the first layer rebuilt instead of cached (w/ CREATED timestamp of 4m ago instead of 15m ago) is a RUN command and that the command text is the same as above
IMAGE CREATED CREATED BY SIZE COMMENT
sha256:5453d7caedb1027c4178baf4737e9fdc608f3d4abb7b7f2cc5baeeb7209bb016 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c rm -rf /tmp/* # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c mamba run -n otter-env pip install /tmp/otter-grader # buildkit 1.43MB buildkit.dockerfile.v0
<missing> 4 minutes ago ADD __otter-grader /tmp/otter-grader # buildkit 68.9MB buildkit.dockerfile.v0
<missing> 4 minutes ago ENV PATH=/root/miniforge3/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin 0B buildkit.dockerfile.v0
<missing> 4 minutes ago ADD tests /autograder/source/tests/ # buildkit 3.64kB buildkit.dockerfile.v0
<missing> 4 minutes ago ADD files* /autograder/source/files/ # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago ADD otter_config.json run_otter.py /autograder/source/ # buildkit 215B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c touch /autograder/submission_metadata.json # buildkit 0B buildkit.dockerfile.v0
<missing> 4 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c dos2unix /autograder/run_autograder /autograder/source/setup.sh && chmod +x /autograder/run_autograder && apt-get update && bash /autograder/source/setup.sh && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/* # buildkit 5.48GB buildkit.dockerfile.v0
<missing> 15 minutes ago ADD setup.sh environment.yml requirements.* /autograder/source/ # buildkit 1.46kB buildkit.dockerfile.v0
<missing> 15 minutes ago ADD run_autograder /autograder/run_autograder # buildkit 219B buildkit.dockerfile.v0
<missing> 15 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c mkdir -p /autograder/source && mkdir -p /autograder/submission && mkdir -p /autograder/results # buildkit 0B buildkit.dockerfile.v0
<missing> 15 minutes ago RUN |1 DEBIAN_FRONTEND=noninteractive /bin/sh -c apt-get update && apt-get install -y curl unzip dos2unix wget && apt-get clean && rm -rf /var/lib/apt/lists/* /var/tmp/* # buildkit 8MB buildkit.dockerfile.v0
<missing> 15 minutes ago ARG DEBIAN_FRONTEND=noninteractive 0B buildkit.dockerfile.v0
<missing> 3 months ago /bin/sh -c #(nop) CMD ["/bin/bash"] 0B
<missing> 3 months ago /bin/sh -c #(nop) ADD file:ebe009f86035c175ba244badd298a2582914415cf62783d510eab3a311a5d4e1 in / 77.9MB
<missing> 3 months ago /bin/sh -c #(nop) LABEL org.opencontainers.image.version=22.04 0B
<missing> 3 months ago /bin/sh -c #(nop) LABEL org.opencontainers.image.ref.name=ubuntu 0B
<missing> 3 months ago /bin/sh -c #(nop) ARG LAUNCHPAD_BUILD_ARCH 0B
<missing> 3 months ago /bin/sh -c #(nop) ARG RELEASE 0B