Docker Community Forums

Share and learn in the Docker community.

How to prevent rebuilding of pulled image?

I am using Travis CI for building and testing my Python project. This project depends on some gentoo-specific python packages, which cannot be installed from PyPi. Therefore I am using gentoo/stage3-amd64 base image with this packages.

Dockerfile:

FROM gentoo/portage:20200420 as portage

FROM gentoo/stage3-amd64:20200420 as toolimg

ARG HOST_USER_UID=1000
ARG HOST_USER_GID=1000

ENV PORTCLEAN_WORK="/work" \
        PORTCLEAN_HOST_USER_UID="${HOST_USER_UID}" \
        PORTCLEAN_HOST_USER_GID="${HOST_USER_GID}" \
        PORTCLEAN_PYTHON_VERSION="3.6" \
        \
        PYTHONFAULTHANDLER=1 \
        PYTHONHASHSEED=random \
        PYTHONUNBUFFERED=1 \
        PYTHONHASHSEED=random \
        \
        PIP_DEFAULT_TIMEOUT=100 \
        PIP_DISABLE_PIP_VERSION_CHECK=1

# Copy the entire portage volume in
COPY --from=portage /var/db/repos/gentoo /var/db/repos/gentoo

COPY ./package.use_dev /etc/portage/package.use/dev
COPY ./package.accept_keywords_dev /etc/portage/package.accept_keywords/dev
COPY ./make.conf_part /etc/portage/make.conf_part
COPY ./setup.sh /setup.sh

RUN /bin/bash /setup.sh \
         && rm --force /setup.sh

WORKDIR "${PORTCLEAN_WORK}"
ENTRYPOINT ["/bin/bash", "-c"]
CMD ["/bin/bash"]

Content of setup.sh:

#!/usr/bin/env bash
set -xeuf -o pipefail


cat /etc/portage/make.conf_part >> /etc/portage/make.conf
rm --force /etc/portage/make.conf_part


# Setting python version
PYTHON_VERSION_UNDERSCORE=$(echo "${PORTCLEAN_PYTHON_VERSION}" \
							| sed --expression "s|\.|_|")
sed \
  --expression "s|PORTCLEAN_PYTHON_VERSION|${PYTHON_VERSION_UNDERSCORE}|" \
  --in-place \
  /etc/portage/make.conf

PYTHON_INTERPRETER_NUMBER=$(eselect python list \
								| grep "${PORTCLEAN_PYTHON_VERSION}" \
								| awk '{print $1}' \
								| sed -e "s|\[||" \
								| sed -e "s|\]||")
eselect python set "${PYTHON_INTERPRETER_NUMBER}"


emerge --quiet=y --verbose=y "app-portage/layman::gentoo"
sed --expression "s|check_official *: *Yes|check_official : No|g" \
	--in-place \
	/etc/layman/layman.cfg
layman \
  --overlays=https://raw.github.com/armoken/gentoo-overlay/master/overlay.xml \
  --add=armoken-overlay \
  --fetch


emerge --quiet=y \
	   --verbose=y \
	   "dev-python/pip::gentoo" \
	   "dev-lang/python:${PORTCLEAN_PYTHON_VERSION}::gentoo" \
	   "app-portage/gentoolkit::gentoo" \
	   "dev-python/poetry::armoken-overlay"


echo "Creating notroot user and group from host"
groupadd --gid "${PORTCLEAN_HOST_USER_GID}" notroot
useradd --no-log-init \
		--create-home \
		--uid "${PORTCLEAN_HOST_USER_UID}" \
		--gid "${PORTCLEAN_HOST_USER_GID}" \
		notroot


mkdir --parents "${PORTCLEAN_WORK}"
chown --recursive notroot:notroot "${PORTCLEAN_WORK}"

Content of .travis.yml:

language: python

os:
  - linux

dist: bionic

services:
  - docker

before_script:
  - /bin/bash ./infra/before_script.sh

script:
  - make build
  - make lint
  - make test

after_success:
  - /bin/bash ./infra/after_success.sh

after_script:
  - /bin/bash ./infra/after_script.sh

Content of before_script.sh:

#!/usr/bin/env bash
set -xeuf -o pipefail

if [[ "${TRAVIS_PULL_REQUEST}" == "false" ]]; then
	echo "${DOCKER_PASSWORD}" \
		| docker login --username "${DOCKER_USERNAME}" \
					   --password-stdin
	make try_pull_testimg
fi

make testimg

if [[ "${TRAVIS_PULL_REQUEST}" == "false" ]]; then
	make push_testimg
fi

make overlay_dirs mounting_overlay_dirs venv_setup

Required content of Makefile:

PORTCLEAN_GIT_COMMIT     := $(shell git rev-parse --short HEAD)
PORTCLEAN_GIT_BRANCH     := $(if $(TRAVIS_BRANCH),$(TRAVIS_BRANCH),$(shell git rev-parse --abbrev-ref HEAD))
PORTCLEAN_TAG            := $(PORTCLEAN_GIT_BRANCH)-$(PORTCLEAN_GIT_COMMIT)
PORTCLEAN_IMG_NAME        = armoken/portclean_test
PORTCLEAN_TAGGED_IMG_NAME = $(PORTCLEAN_IMG_NAME):$(PORTCLEAN_TAG)
PORTCLEAN_LATEST_IMG      = $(PORTCLEAN_IMG_NAME):latest

PORTCLEAN_UID := $(shell id --user)
PORTCLEAN_GID := $(shell id --group)


testimg:
	@echo ">>> Start testimg building <<<"
	cd ./infra/contrib/toolimg \
		&& docker build \
				--build-arg=HOST_USER_UID="${PORTCLEAN_UID}" \
				--build-arg=HOST_USER_GID="${PORTCLEAN_GID}" \
				--tag=${PORTCLEAN_TAGGED_IMG_NAME} \
				--tag=${PORTCLEAN_LATEST_IMG} \
				.


push_testimg:
	@docker push ${PORTCLEAN_TAGGED_IMG_NAME}
	@docker push ${PORTCLEAN_LATEST_IMG}


try_pull_testimg:
	-@docker pull ${PORTCLEAN_LATEST_IMG}

Build logs:

$ /bin/bash ./infra/before_script.sh

+[[ false == \f\a\l\s\e ]]

+docker login --username [secure] --password-stdin

+echo [secure]

WARNING! Your password will be stored unencrypted in /home/travis/.docker/config.json.

Configure a credential helper to remove this warning. See

https://docs.docker.com/engine/reference/commandline/login/#credentials-store

Login Succeeded

+make try_pull_testimg

latest: Pulling from [secure]/portclean_test

5b42289cfbaf: Pulling fs layer

984adaf49b9e: Pulling fs layer

4c14678aa7bf: Pulling fs layer

d34fe3a3427a: Pulling fs layer

7e113af84cff: Pulling fs layer

a5af241ee9b0: Pulling fs layer

26d326ac3255: Pulling fs layer

d34fe3a3427a: Waiting

7e113af84cff: Waiting

a5af241ee9b0: Waiting

26d326ac3255: Waiting

4c14678aa7bf: Verifying Checksum

4c14678aa7bf: Download complete

d34fe3a3427a: Verifying Checksum

d34fe3a3427a: Download complete

7e113af84cff: Verifying Checksum

7e113af84cff: Download complete

a5af241ee9b0: Verifying Checksum

a5af241ee9b0: Download complete

984adaf49b9e: Verifying Checksum

984adaf49b9e: Download complete

5b42289cfbaf: Download complete

26d326ac3255: Verifying Checksum

26d326ac3255: Download complete

5b42289cfbaf: Pull complete

984adaf49b9e: Pull complete

4c14678aa7bf: Pull complete

d34fe3a3427a: Pull complete

7e113af84cff: Pull complete

a5af241ee9b0: Pull complete

26d326ac3255: Pull complete

Digest: sha256:476504991ec996015187fa9d66253e10050b082ed6f3a18e07ae8a50f23d98f8

Status: Downloaded newer image for [secure]/portclean_test:latest

+make testimg

>>> Start testimg building <<<

cd ./infra/contrib/toolimg \

&& docker build \

--build-arg=HOST_USER_UID="2000" \

--build-arg=HOST_USER_GID="2000" \

--tag=[secure]/portclean_test:master-077003b \

--tag=[secure]/portclean_test:latest \

.

Sending build context to Docker daemon  8.704kB

Step 1/14 : FROM gentoo/portage:20200420 as portage

20200420: Pulling from gentoo/portage

e2334dd9fee4: Pulling fs layer

43b249293ec0: Pulling fs layer

e2334dd9fee4: Verifying Checksum

e2334dd9fee4: Download complete

e2334dd9fee4: Pull complete

43b249293ec0: Verifying Checksum

43b249293ec0: Download complete

43b249293ec0: Pull complete

Digest: sha256:d12ce083f5dc707671f3909110d51800482c2e30720e2f81034b8905afd0c504

Status: Downloaded newer image for gentoo/portage:20200420

---> fdea6c7ccdf8

.... A lot of useless logs ....

Step 14/14 : CMD ["/bin/bash"]

---> Running in 51d1285bbdb5

Removing intermediate container 51d1285bbdb5

---> 8729a23502fb

Successfully built 8729a23502fb

Successfully tagged [secure]/portclean_test:master-077003b

Successfully tagged [secure]/portclean_test:latest

+[[ false == \f\a\l\s\e ]]

+make push_testimg

The push refers to repository [docker.io/[secure]/portclean_test]

79955140ac96: Preparing

46d8a7eb953b: Preparing

27efef6e72ad: Preparing

09d70d5d336b: Preparing

569b9d1625d9: Preparing

9fa38ace824e: Preparing

4b9197d94da9: Preparing

9fa38ace824e: Waiting

4b9197d94da9: Waiting

569b9d1625d9: Pushed

46d8a7eb953b: Pushed

09d70d5d336b: Pushed

27efef6e72ad: Pushed

4b9197d94da9: Layer already exists

79955140ac96: Pushed

9fa38ace824e: Pushed

master-077003b: digest: sha256:60b8b14b8ee2c5eb41c7501f7f38ada9b9998faf165b93930d64e8fd56323d05 size: 1783

The push refers to repository [docker.io/[secure]/portclean_test]

79955140ac96: Preparing

46d8a7eb953b: Preparing

27efef6e72ad: Preparing

09d70d5d336b: Preparing

569b9d1625d9: Preparing

9fa38ace824e: Preparing

4b9197d94da9: Preparing

9fa38ace824e: Waiting

4b9197d94da9: Waiting

46d8a7eb953b: Layer already exists

27efef6e72ad: Layer already exists

569b9d1625d9: Layer already exists

79955140ac96: Layer already exists

09d70d5d336b: Layer already exists

4b9197d94da9: Layer already exists

9fa38ace824e: Layer already exists

latest: digest: sha256:60b8b14b8ee2c5eb41c7501f7f38ada9b9998faf165b93930d64e8fd56323d05 size: 1783

Can you describe your problem a bit more? How to prevent rebuilding of pulled image? gives quite a lot of room for interpretation.

What do you want?
What does actually happen?

I would put my money on “cache hits due to ARGs”, see: https://docs.docker.com/engine/reference/builder/#impact-on-build-caching

I fixed it by adding –cache-from=${PORTCLEAN_LATEST_IMG} argument to the image build command. But I am not understand why it doesn’t required when I am building image locally.