Docker Community Forums

Share and learn in the Docker community.

Docker-compose not seeing environment variables on the host


(Andyburke) #1

Expected behavior

docker-compose.yml should interpret environment variables, eg:

test:
    build: .
    dockerfile: Dockerfile
    environment:
        - HOST_HOSTNAME=${HOSTNAME}

should result in HOST_HOSTNAME being set to the host’s hostname in the container.

Actual behavior

> echo $HOSTNAME
foo
> docker-compose up
WARNING: The HOSTNAME variable is not set. Defaulting to a blank string.
...

Information

  • a reproducible case if this is a bug, Dockerfiles FTW

Dockerfile:

FROM alpine:3.3

RUN apk --no-cache add coreutils 

CMD [ "env" ]

docker-compose.yml:

test:
    build: .
    dockerfile: Dockerfile
    environment:
        - HOST_HOSTNAME=${HOSTNAME}

Expected output:

test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=63d376bcaa02
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root

Actual output:

test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=63d376bcaa02
test_1  | HOST_HOSTNAME=
test_1  | HOME=/root
  • host distribution and version ( OSX 10.10.x, OSX 10.11.x, Windows, etc )

OSX:
10.11.4 (15E65)

Docker for Mac:
Version 1.11.1-beta11 (build: 6974)
37559e5f6acd56a4810963acc7001e88f2d88017

docker-compose:
docker-compose version 1.7.1, build 0a9ab35


(Dave Tucker) #2

Thanks for the report.
Unfortunately I’m unable to reproduce…

$ export HOSTNAME=foo
$ docker compose up
Creating foo_test_1
Attaching to foo_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=125412e7116e
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root
foo_test_1 exited with code 0

The only thing that I can think of is that variable HOSTNAME isn’t exported to the child process, in this case, docker-compose.
For example, the following won’t work…

HOSTNAME=foo
docker-compose up

(Andyburke) #3

I see. So the issue is that even though the variable is set in my shell, I have to export it? This is confusing, I haven’t seen that kind of behavior before. Eg:

> set | grep HOSTNAME
HOSTNAME=foo
> docker-compose up
WARNING: The HOSTNAME variable is not set. Defaulting to a blank string.
Starting dockercomposeenvvariables_test_1
Attaching to dockercomposeenvvariables_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=2d4d5e216cb6
test_1  | HOST_HOSTNAME=
test_1  | HOME=/root
dockercomposeenvvariables_test_1 exited with code 0
> export HOSTNAME=${HOSTNAME}
> docker-compose up
Recreating dockercomposeenvvariables_test_1
Attaching to dockercomposeenvvariables_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=9cb89136063e
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root
dockercomposeenvvariables_test_1 exited with code 0

Is this intended behavior? I’ve never had to export variables that were already set in my shell before.


(Alexandre) #4

That’s pretty standard. Everything that is not EXPORTed will not be passed to other processes. If you’d like to know more:

  • Note that this is not docker-compose related, but really a shell-specific behavior. Indeed, it is the shell that’ responsible for passing or not variables to the sub-process’s environment, and it this case it does not because the variable was not marked for EXPORT.

(Andyburke) #5

Ok, consider this closed: I’ve learned something new.


(Dalevisser) #6

I saw this error message. In my case, the confusion was caused by the fact that I was using .env to set the environment for Docker compose, and I was running docker-compose commands in a child folder of the folder holding docker-compose.yml and .env.


(Bertoldi) #7

Hi, I’m having the same problem, there is something I really don’t get about variable substitution.
I tried the example above and everything is fine if my env variable is “HOSTNAME”, but I’m not able to interpolate new variables in the compose file, for example “FOO”:

# example/Dockerfile
FROM alpine
# example/docker-compose.yml
version: '3'

services:
  test:
    build: .
    environment:
      FOO: $FOO
    command: echo $FOO
# Terminal
$ export FOO=bar
$ sudo docker-compose up
WARNING: The FOO variable is not set. Defaulting to a blank string.
Starting example_test_1 ... 
Starting example_test_1 ... done
Attaching to example_test_1
test_1  | 
example_test_1 exited with code 0

(Bertoldi) #8

I figured it out, I’m posting this in case someone might need it.
My problem was that the environment variables are not preserved by default when sodoing.
So, in order for compose variable substitution to work, when the user is not added to docker group, you must add the -E flag to sudo.
For the example above:

$ export FOO=bar
$ sudo -E docker-compose up
Creating network "example_default" with the default driver
Creating example_test_1 ... 
Creating example_test_1 ... done
Attaching to example_test_1
test_1  | bar
example_test_1 exited with code 0

ps. docker-compose config really helped me debug this, it shows in the shell the result of the variables substitutions in docker-compose.yml.


(José Luis) #10

I had this problem a few minutes ago, i had to close the terminal and open new one; all is well now.


(Mmoo9154) #11

FWIW, I’m having this problem today. I’m using docker-compose version 1.22.0, build f46880fe on Ubuntu 18.04.1.

The env variable in my docker-compose.yml is ${PWD}, and I’ve confirmed the variable exported. (I also quit and restarted the terminal “just in case”.

When I sudo docker-compose run --service-ports meteor I get the following output:

WARNING: The PWD variable is not set. Defaulting to a blank string.
Starting todos_mongo_1 ... done
ERROR: Cannot create container for service meteor: b'create .: volume name is too short, names should be at least two alphanumeric characters'

I did not see any suggestions above that fix this problem. Is it a bug in this release of docker-compose?


(Dean Kayton) #12

I am having the following problem:

The relevant bit of the compose file is:

version: "3.7"
services:
  example:
    ...
    environment:
      ...
      HOST_UID: $HOST_UID
      HOST_GID: $HOST_GID
    ....
The following do perform variable substitution:
HOST_UID=1030 HOST_GID=1029 docker-compose -f docker-compose-prod.yml config
The following do not perform variable substitution:
HOST_UID=1030
HOST_GID=1029
docker-compose -f docker-compose-prod.yml config

This is as expected right?

Interestingly, when I change the compose spec to be the following:

version: "3.7"
services:
  example:
    ...
    environment:
      ...
      HOST_UID: $UID
      HOST_GID: $GROUPS
      SOMEOTHER: $SOMEOTHER
      PWD: $PWD
    ....

then the following behaviour is observed in a brand new shell:

$ SOMEOTHER=123
$ echo $SOMEOTHER
123
$ echo $UID
1000
$ echo $GROUPS
1000
$ echo $PWD
/home/.../src/...
$ docker-compose -f docker-compose-prod.yml config

It gives back:

    environment:
      HOST_GID: ''
      HOST_UID: '1000'
      PWD: /home/.../src/...django-backend
      SOME_OTHER: '123'

So here we see that an example of a variable that compose can see is PWD. What is the name for this class of variables? It would be nice if compose could also see all the default variables that are set in a brand new shell. Why is this not the case?

Another interesting observation. The behaviour is odd even when exporting the variables (again in a brand new shell)!

$ SOMEOTHER=123
$ echo $SOMEOTHER
123
$ echo $UID
1000
$ echo $GROUPS
1000
$ echo $PWD
/home/.../src/...
$ # The above is same as before, below we now export these variables
$ export SOMEOTHER
$ export UID
$ export GROUPS
$ # No need to export PWD
$ docker-compose -f docker-compose-prod.yml config

It gives back:

WARNING: The GROUPS variable is not set. Defaulting to a blank string.
...
 environment:
      ...
      HOST_GID: ''
      HOST_UID: '1000'
      PWD: /home/.../src/...
      SOME_OTHER: '123'

Why is the GROUPS variable not set!? Something is fishy here.

Finally:

docker-compose -v
docker-compose version 1.22.0, build f46880fe

docker -v
Docker version 18.06.1-ce, build e68fc7a