Combining docker-compose override files with mutliple .env files

General
1

Hi,

I am trying to optimise my setup for different environments using multiple docker-compose..yml files and also multiple ..env files. But I am not successful.

My docker-compose.yml is my base compose file that is used on every environment. For each environment I have defined an override file such as docker-compose.linode.yml. I will just list ports for demonstration purposes and the parts regarding env variables as that is the area where I can’t get it to work.

## docker-compose.yml

services:
  myapp:
    ports:
      - 8090:8090
    environment:
      FA_SERVER: ${FA_SERVER}

## docker-compose.linode.yml
## override ports & .env files

services:
  myapp:
    ports:
      - 80:8090
    env_file:
      - .env
      - .linode.env
    environment:
      FA_SERVER: ${FA_SERVER}

## base .env
FA_SERVER=LOCAL
...

##  .linode.env (override base .env)
FA_SERVER=LINODE
...

Locally I will just spin it up with (no problems here)

docker-compose up -d

FA_SERVER will evaluate to LOCAL.

Then on my Linode VPS I would do:

docker-compose -f docker-compose.yml -f docker-compose.linode.yml up -d

FA_SERVER still evaluates to LOCAL isntead of LINODE.

So my expectation is, that through the docker-compose.linode.yml the two .env files are evaluated with the one listed below having priority (i.e. .linode.env) as per the documentation:

https://docs.docker.com/compose/compose-file/compose-file-v3/#env_file

Keep in mind that the order of files in the list is significant in determining the value assigned to a variable that shows up more than once . The files in the list are processed from the top down. For the same variable specified in file a.env and assigned a different value in file b.env , if b.env is listed below (after), then the value from b.env stands. For example, given the following declaration in docker-compose.yml

and https://docs.docker.com/compose/environment-variables/#the-env_file-configuration-option

When you set the same environment variable in multiple files, here’s the priority used by Compose to choose which value to use:

  1. Compose file
  2. Shell environment variables
  3. Environment file
  4. Dockerfile
  5. Variable is not defined

The idea is to simplify the commands to spin up the containers with compose and not having to use also arguments for the env files. I also believe docker-compose up does not support cli arguments for env files, while docker-compose run does not support cli arguments to specify multiple compose files.

Thanks for help in advance!

2

I see why it is confusing but the env_file in the docker compose file is for setting variables in a container. Just like environment. You could do it using docker like docker run -e FA_SERVER=LINODE or docker run --env-file linode.env. Those variables will not be available in the docker compose file.

.env is for docker compose, whcih means it will be interpreted by docker compose and you can use the variables in the yaml file. Since your .env contains LOCAL, this is the value of ${FA_SERVER} in docker-compose.linode.yml.

The list you find

refers that you can define variables for a container multiple ways.

let’s say you have a .env file:

source=.env

and defined in the shell:

export source=shell

compose file

services:
  ubuntu:
    image: ubuntu:20.04
    stdin_open: true
    tty: true
    env_file:
      - custom.env
    environment:
      source: compose

Then run

docker-compose up -d
docker-compose exec ubuntu env | grep source

You would see

source=compose

Then you remove the environment section from the compose file and keep the env_file section, run docker exec again to see

source=custom.env

None of these values came from the .env file neither from the shell because you defined it in the docker compose either using the environment section or the env_file section.

Now remove the env_file section from the compose file, and you won’t see source defined in the container becuse the shell variables and the content of the .env file will not be set in the containers.

Put the environment section back this way:

services:
  ubuntu:
    image: ubuntu:20.04
    stdin_open: true
    tty: true
    env_file:
      - custom.env
    environment:
      source: ${source}

Now you will see source=shell in the output because the shell has higher precedence (2) than the .env file (3).
Now delete the shell variable:

unset source

Run docker-compose up and docker exec again to see this:

source=.env

If you don’t have .env file but you have source set in the Dockerfile, then the container see that value like source=dockerfile.

Let’s call .env and the shell external sources. I don’t know if anyone calls it that way but I will. So if you use these external sources, then if you have a variable accidentaly or intentionally defined in. the shell, it overrides the values in the .env file. Even if you could have multiple .env files.

You can use docker-compose.linode.yml to set the variables without any external source, or create shell scripts for the different setups to set different variables in each case.

3

Thanks @rimelek,

I just saw your answer now but I had already been typing everything below over the course of the afternoon, so I am not gonna change it, even though it might be kind of confusing. :grinning:

I figured it out, at least partly.

Like the docs say, the order of evaluation is

  1. Compose file
  2. Shell environment variables
  3. Environment file
  4. Dockerfile
  5. Variable is not defined

I also found this:

To summarize, .env file variables are not passed to the docker container at runtime. The .env file is used for variable substitutions in config files only
The env_file block on the other hand passes env vars from a file to the container at runtime, serving the same purpose as the environment block or docker run -e .

So basically that means, .env should be used for variable substitutions in the compose files, for instance for setting path for volumes. Of course it is still possible to pass variables from .env into containers by using this substitution in the environment: block. On the other hand, this block will only substitute from .env and never from other env files specified with env_file:.

From my tests I “rewrote” the list from the docs as follows:

  1. override compose file:
    1.1 hardcoded (VAR: “override-value”)
    1.2. VAR: ${VAR} → always from base .env regardless of env_file: setting*
  2. base compose file:
    2.1. hardcoded (VAR: “base-value”)
    2.2. VAR: ${VAR} → always from base .env regardless of env_file: setting*
  3. “override” env file like .linode.env: this only passed into the containers if the variable is not set in any compose file in the environment: part.

Scenarios
So I spent the Sunday afternoon setting up a test project to figure out different scenarios:

  • App config variables shared across apps/containers and environments:
    • just define them in the base .env and pass them to containers including .env in the env_file: option, then these will be included in all containers.
  • App config variables that are shared across environments (but not across apps/containers), e.g. like EMAIL_SUBJECT that is only necessary for an “emailing container” or API_URL for a container that requests data from an external api:
    • Selectively define VAR: ${VAR} for each container (will read from .env) and don’t include .env in the env_file: option.
  • Variables that are shared between different apps/containers (but not across environments), e.g. MYSQL_HOST if more than one container use the same database but a productive environment might use a managed DB host:
    • Put them in local.env and .linode.env with their different values and just include them for each container with the env_file: option. (In my case, I could use this, since for instance multiple apps/containers all connect to the same database container. WARNING: you might end up with secrets in containers where they are not used, this could become a security issue). I will have to check how Docker Secrets work.

What are the options?

  • setting the hardcoded variable in docker-compose.linode.yml under environment:. This will will override the var from docker-compose.yml. Hardcoded, since if I used ${VAR} it would read from .env what I don’t want, since this is different for a different einvironment.

  • not setting the variable in neither docker-compose.yml nor docker-compose.linode.yml under environment: and only set it in .local.env and .linode.env. Though like this, the variables will be passed into all containers using this file. Maybe it could even become a security risk (if container A gets compromised the attacker could then see secrets for container B).

  • I could go further and use .<app>.<env>.env so for each app/container and environment combination it’s own file. But let’s say there was 3 apps and 3 environments, that would already be 9 files. So that does not make sense to me

  • Another option: use different variable names in .env for the same thing but different environments (like FA_SERVER_DEV=LOCAL and FA_SERVER_PROD=LINODE and then using them in the corresponding compose files for substitution (though I don’t think that really fits the idea of having different env files for different environments):

    • docker-compose.override.yml: FA_SERVER=${FA_SERVER_DEV}
    • docker-compose.linode.yml: FA_SERVER=${FA_SERVER_PROD}

For now I will just keep it like this:

  • Most app config variables are the same over all environments, so I leave them in the .env file and they are selectively passed into the different containers through substitution in the environment: blocks.
  • Some few app config variables that are different on the environments (like secrets) I have put in .<env>.env and they will be included in the corresponding compose override files.

Final Thoughts
While the whole thing with inheritance, extending and overriding variables (and other parts in compose files) is great to reduce redundancy, it can also increase complexity to a point, where I am not sure any more what I am actually doing :smile:

What I probably find most confusing is that when using an override compose file with variable substitution in the environment: block, this will still use the .env. Maybe there should be an option to define the source for variable substitution as well.