Perform environment variable normalization on docker-compose file without running the container

General
1

Good afternoon everyone,

I’m trying to write a script that will (as much as possible) automate the process of converting our docker-compose.yml file into a docker-compose file that can work on top of BalenaCloud. The issue with this is that BalenaCloud works with a smaller feature-set than native docker-compose, meaning that all environment variables (we use a lot of them, and we use the different kinds of substitutions like errors and default values etc) will have to be normalized before submitted to BalenaCloud.

I read up about docker-compose config with the --env-file option, and that does normalize the environment variables as needed. The issue seems to be that it also provides (as the documentation suggests) a much more granular docker-compose file, the directives of which are not accepted by the balena CLI.

Is there any other alternative to normalize docker-compose environment variables found in a docker-compose.yml file?

Thanks!

2

I’m not sure what exactly you would like to do and what you mean by “normalize”, but have you tried

docker compose config

?

Usage:  docker compose config [OPTIONS] [SERVICE...]

Parse, resolve and render compose file in canonical format
3

I’m a big fan of docker compose config since it show how docker will “merge” and “render” yaml files. Very useful.

4

Thanks for replying!

By normalizing I mean converting ${USER_PASSWORD:-secret} to secret depending on its presence on the environment variable file, or any kind of environment variable directive, when an environment variable file is passed.

I tried using docker compose config but the issue is that it also converts other directives into more precise directives:


    ports:
      - mode: ingress
        target: 80
        published: "80"
        protocol: tcp
      - mode: ingress
        target: 443
        published: "443"
        protocol: tcp

something which is not directly supported in BalenaCloud.

I thought of writing a custom script for this but there’s a lot of logic surrounding it, with default values, error values, etc…

So basically I’d like to resolve environment variables into their actual final form, without everything else docker compose config outputs.

Thanks!

EDIT: on my original post I meant to say docker compose config, not format. Brainfart.

5

The only perfect way could be using the same library as Docker Compose uses, but in a simple case you could use envsubst in a shell

envsubst < docker-compose.yml

If you want to read dotenv files as well, you can try this:

echo "$(export $(cat .env) && envsubst < docker-compose.yml)"

or

echo "$(set -o allexport && source .env && envsubst < docker-compose.yml)"

The problem with these is that docker compose config supports a value like this in the dotenv file:

TEST=2 '

where the apostrophe is part of the string, while the shell could not interpret it.

6

The issue with envsubst, because I tried it, is that it does not understand the directives that docker-compose does here: Syntax for environment files in Docker Compose | Docker Docs

It does understand the simple ones, but ${USER_PASSWORD:-secret} will not be treated correctly.

7

I did find the library for it in Go I think: compose/pkg/compose/envresolver.go at ff20b641c7f1cce588faea476fbd1a6fdd953b36 · docker/compose · GitHub

I am thinking that the worst case scenario is to repackage/rewrite it in python or something like that.

8

I made this which worked

echo "$(export $(cat .env) && eval "echo $(printf 'cat "$(<< EOF\n%s\nEOF\n)"'  "$(cat docker-compose.yml)")")"

It has many subshells. Maybe it could be optimized. It is based on the fact that a simple shell can interpret the syntax and if it is converted to an “echo” command, you just need to “eval” it.

I got some ideas from this post:

but my original idea came from here:

where I didn’t read all the comments, so you could read it hoping you find something useful.

9

You could try if the output of docker compose config can be piped into composeverter somehow to convert it back to the short syntax. I assume it provides a cli interface with the same functionality as https://www.composeverter.com.

This could also be a nice feature request for docker compose actually.

There is a --no-normalization argument for docker compose config, but I have no idea what it does, and somehow fail to see a difference in the output.

10

--no-normalize removes some defaults like the network definition for a service when it is not defined in the file. It could also throw an error without --no-consistency when the default network is listed in the service block, but not in the top-level network block, which seems like a bug.

1 Like