Runtime environment variable requirements in build stage

Hi Docker Community,

I have been wondering about an inconvenience that I have encountered for some time, and I was wondering how the community is currently handling this.

Through variable substitution, we can make sure that some variables are mandatory so that the runtime application will at least have the minimal requirements to run. Without extra configuration, from this same docker compose definition, we can also trigger a build using docker compose build. For example, with the following minimal definition:

services:
  test: 
   image: scratch
   environment:
    - TEST=${TEST:?}

*This example does not make much sense, but it shows the problem :slight_smile: *

Now, in the build phase, it would seem to not make much sense to define runtime variables. However, docker compose seems to still check the environments property for mandatory variables. In our CI/CD, the build phase is not assigned to any environment, and thus not all mandatory environment variables are being passed in the build phase.

How we fixed this is to pass along empty values by default. This, however, feels very cumbersome, and I was wondering if any has experience with a nicer solution?


Solution I am aware of

  • Create a separate docker compose file for the build phase (docker-compose.build.yml) → This would work, but an extra compose file, which preferable should be extended (using the -f options) for the actual runtime file(s), would add a lot more complexity. Especially already having multiple docker compose files (live/production, local, base, etc).

I have never seen anyone in corporate context that actually used docker compose inside pipelines to create images.

During the last years the pipelines I have seen used docker build, buildkit, kaniko or buildah to build oci images, usually followed by a tool like Synk or Trivi to scan the created image for vulnerability before the image gets pushed to a private registry.

I agree with @meyay so I will respond only focusing on Docker Compose and not CI/CD.

We use Docker compose mainly for local development or on servers where don’t have the resources to use something else or run that on a single host (Swarm, Kubernetes). So we usually have all the required variables. Docker Compose handles variables the same way regardless what subcommand you use. Even docker compose ps would fail without setting the required variables In rare cases this behavior can be inconvenient, but I don’t even remember when I had any problem with it. I know I had, but I don’t know why and when.

Using your example run the following command:

docker compose config

It will cpmplain about the missing variable. If you run this way:

TEST=1 docker compose config

It will show something like this:

services:
  test:
    environment:
      TEST: "1"
    image: scratch
    networks:
      default: null
networks:
  default:
    name: vars_default

As you can see it will generate the configuration file replacing the variable with its value. I guess it is because this way Docker does not have to be able to parse different kind of yaml files that it supports. Instead of that, it converts the yaml to a normalized form. Using the config subcommand you can see the config without normalizing it:

TEST=1 docker compose config --no-normalize
services:
  test:
    environment:
      TEST: "1"
    image: scratch

but it still replaces the variable (interpolation). For debugging purposes you can disable that too

TEST=1 docker compose config --no-normalize --no-interpolate
services:
  test:
    environment:
      TEST: ${TEST:?}
    image: scratch

but it doesn’t make sense with other commands. The main purpose of these environment variables is that you can use the compose file as a template and then render that template. Not just one part but the entire file. If you want those variables to be required by the running container only, you need to build that checking into the entrypoint or the command. Let’s see an example to the command:

services:
  test:
   image: bash
   environment:
     - TEST
   command:
    - -c
    - echo $${TEST:?}
    # Using double dollars is required so compose will not recognize it as variable.
docker compose config
services:
  test:
    command:
    - -c
    - echo $${TEST:?}
    environment:
      TEST: ""
    image: bash
    networks:
      default: null
networks:
  default:
    name: vars_default

Notice that TEST: "" part.
Now run this

TEST=1 docker compose config
services:
  test:
    command:
    - -c
    - echo $${TEST:?}
    environment:
      TEST: "1"
    image: bash
    networks:
      default: null
networks:
  default:
    name: vars_default

Let’s run the container

docker compose up
[+] Running 1/0
 ⠿ Container vars-test-1  Created                                                                                                                       0.0s
Attaching to vars-test-1
vars-test-1  | bash: line 1: TEST: parameter null or not set
vars-test-1 exited with code 127
TEST=1 docker compose up
[+] Running 1/0
 ⠿ Container vars-test-1  Recreated                                                                                                                     0.0s
Attaching to vars-test-1
vars-test-1  | 1
vars-test-1 exited with code 0

Example to the entrypoint:

entrypoint.sh

#!/usr/bin/env bash

set -eu -o pipefail

: ${TEST:?}


if command -v "$1" &>/dev/null; then
  exec "$@"
else
  exec bash "$@"
fi

Dockerfile

FROM bash

COPY entrypoint.sh /custom-entrypoint.sh

RUN chmod +x /custom-entrypoint.sh

ENTRYPOINT ["/custom-entrypoint.sh"]

compose.yml

services:
  test:
   build: .
   environment:
     - TEST
   command:
    - -c
    - "echo $${TEST}"

The result is the same. Of course the required entrypoint will be different for every application.

Thank you @meyay and @rimelek both for the explanations!

The only reason why I was using the docker compose for the build phase, is to make sure we use the same tag that is used in runtime.

@rimelek It seems that docker-compose config --no-interpolate does give me a bit more freedom. The --images option exists but sadly does not listen to the --no-interpolate options and validates the config. I can extract the image using the --format json output:

docker-compose config --no-interpolate {service} --format json | jq -r '.services | .. | select(.image? != null) | .image'

But this is also not a guarantee, looking at Allow turning off error checking in the config command · Issue #7203 · docker/compose · GitHub. It seems that --no-interpolate seems to be not stable enough.

I guess adding the explicit docker tag is my safest option for now.

Well, I am not sure that is a bug, since using --images would indicate you want to see the actual images, not a part of the yaml, but even if it is expected, I found an actual bug when I tested that option, so I wouldn’t rely on it.

It looks like the interpolation happens in the volume paths even with --no-interpolate is used, but the variable is not removed either. So you might be right with the other issue as well.

It seems you are right about that. I will report what I found. I have reported an error before regarding interpolation and also fixed it, so maybe I will look into this issue too.