How can I expand a variable within a COPY command in the Dockerfile?

Docker Hub
,
1

I have a Dockerfile:

FROM nginx
ARG from_url
ENV REFERENCE_IMAGE_URL ${from_url}
RUN echo "Going to use reference image $REFERENCE_IMAGE_URL"
COPY --from=${REFERENCE_IMAGE_URL} /staticfiles /usr/share/nginx/html/static

I have a make target to use it:

nginx-dist:
	@echo "Using DOCKER_IMAGE_URL=$(DOCKER_IMAGE_URL)"
	@echo "Using Docker version $(shell docker --version)"
	docker build -t prior_auth_nginx:latest -f nginx.Dockerfile . --build-arg from_url=$(DOCKER_IMAGE_URL)

When I run this make target, I get the following:

Using DOCKER_IMAGE_URL=<redacted-url>:<redacted-tag>
Using Docker version Docker version 20.10.2, build 20.10.2-0ubuntu1~18.04.3
docker build -t <redacted>:latest -f nginx.Dockerfile . --build-arg from_url=<redacted-url>:<redacted-tag>
Sending build context to Docker daemon  21.73MB

Step 1/5 : FROM nginx
 ---> 08b152afcfae
Step 2/5 : ARG from_url
 ---> Using cache
 ---> 3abec8f00eec
Step 3/5 : ENV REFERENCE_IMAGE_URL ${from_url}
 ---> Running in 29487f30569c
Removing intermediate container 29487f30569c
 ---> 59020c91dc63
Step 4/5 : RUN echo "Going to use reference image $REFERENCE_IMAGE_URL"
 ---> Running in 343289f7443c
Going to use reference image <redacted-url>:<redacted-tag>
Removing intermediate container 343289f7443c
 ---> d77a60646efa
Step 5/5 : COPY --from=${REFERENCE_IMAGE_URL} /staticfiles /usr/share/nginx/html/static
invalid from flag value ${REFERENCE_IMAGE_URL}: invalid reference format: repository name must be lowercase
make: *** [nginx-dist] Error 1

Based on the documentation (Dockerfile reference | Docker Documentation), I see that COPY is supposed to accept variable expansion. As you can see from the output above, it does not. What am I doing wrong here?

2

The <redacted> things make it a bit hard to get the full picture. So, just to be sure, referring to the actual error message repository name must be lowercase: is the value lowercase? (Maybe it’s just that the error message does not show the substituted value, but shows the original code?)

Also, given the variable’s name, should we assume REFERENCE_IMAGE_URL is some URL including https://? If true, then I’ve never seen that being used in COPY --from (but I am no expert). I guess you tested without the variable, with the hardcoded value?

Documentation for COPY --from I see:

Optionally a name can be given to a new build stage by adding AS name to the FROM instruction. The name can be used in subsequent FROM and COPY --from=<name> instructions to refer to the image built in this stage.

…and:

Optionally COPY accepts a flag --from=<name> that can be used to set the source location to a previous build stage (created with FROM .. AS <name> ) that will be used instead of a build context sent by the user. In case a build stage with a specified name can’t be found an image with the same name is attempted to be used instead.

So, I guess Compose falls back to that very last sentence. But: should that support a URL like https://example.com/something rather than, say, simply repo-name/image-name? (I don’t know.)

3

Sure, the redacted bit does make that hard to troubleshoot. Thank you for pointing it out.

I can verify that the $REFERENCE_IMAGE_URL is in fact all lowercase in the echo statement (4/5).

Also, given the variable’s name, should we assume REFERENCE_IMAGE_URL is some URL including https://?

It is a URL, with a tag at the end of it, but without https:// at the front. It is the equivalent of some.aws.account.url/organization/application:tag (without protocol at the beginning).

4

Right?

If not:

I guess that should be okay:

https://github.com/containerd/containerd/blob/main/reference/docker/reference.go#L20-L40

(Especially as, if the above code is used, the actual error message is raised because the lower-cased value does match the above, whereas the original value does not. Hence some uppercase is present in whatever value is validated.)

So, indeed maybe no variable substitution is taking place. On the other hand, the $, { and } from ${REFERENCE_IMAGE_URL} should be rejected, even when lowercased? :thinking:

5

I feel like I did something like make the variable name lowercased, but it just gave me the same error. The way it echos “invalid from flag value ${REFERENCE_IMAGE_URL}: invalid reference format: repository name must be lowercase” suggests to me that it’s not being processed and replace with the relevant variable. I’m really not sure here.

6

I can verify that it does not have anything uppercase in it, though for proprietary info purposes I cannot put the relevant URL there.

7

I have the same issue, here is a sample Dockerfile to reproduce the bug:

ARG IMAGES_PHP_VERSION="7.4.30"

FROM php:${IMAGES_PHP_VERSION}-apache-bullseye

# Fails:
COPY --from=php:${IMAGES_PHP_VERSION}-cli /usr/local/bin/php-cgi /php-cgi

# Works:
# COPY --from=php:7.4.30-cli /usr/local/bin/php-cgi /php-cgi

When building (docker build .), I get:

Step 3/3 : COPY --from=php:${IMAGES_PHP_VERSION}-cli /usr/local/bin/php-cgi /php-cgi
invalid from flag value php:${IMAGES_PHP_VERSION}-cli: invalid reference format

Also tried some variants:

ARG IMAGES_PHP_VERSION="7.4.30"

ARG CLI_IMAGE_TAG="${IMAGES_PHP_VERSION}-cli"
ARG CLI_IMAGE_NAME_WITH_TAG="php:${IMAGES_PHP_VERSION}-cli"

FROM php:${IMAGES_PHP_VERSION}-apache-bullseye

# Fails:
COPY --from=php:${CLI_IMAGE_TAG} /usr/local/bin/php-cgi /php-cgi

# Fails:
COPY --from=${CLI_IMAGE_NAME_WITH_TAG} /usr/local/bin/php-cgi /php-cgi

My use case is that I want to make sure I grab a file from “php” image of the exact same version (without typing it twice which is error-prone).

8

Seems the --from parameter doesn’t support interpolation.

It works, if you use a multi stage build:

ARG IMAGES_PHP_VERSION="7.4.30"
FROM php:${IMAGES_PHP_VERSION}-cli as php

FROM php:${IMAGES_PHP_VERSION}-apache-bullseye

COPY --from=php /usr/local/bin/php-cgi /php-cgi
9

Thanks for the tip, I hadn’t realized I could simple use an empty FROM foo-${VAR} AS some_name :+1:

I’ve found the GitHub issue for the missing interpolation: Allow to use environment variables in COPY --from · Issue #34482 · moby/moby · GitHub
It looks like it’s not a real bug and won’t be “fixed”: but now, at least, I have an acceptable workaround!