Docker-compose not seeing environment variables on the host

Docker Desktop
1

Expected behavior

docker-compose.yml should interpret environment variables, eg:

test:
    build: .
    dockerfile: Dockerfile
    environment:
        - HOST_HOSTNAME=${HOSTNAME}

should result in HOST_HOSTNAME being set to the host’s hostname in the container.

Actual behavior

> echo $HOSTNAME
foo
> docker-compose up
WARNING: The HOSTNAME variable is not set. Defaulting to a blank string.
...

Information

  • a reproducible case if this is a bug, Dockerfiles FTW

Dockerfile:

FROM alpine:3.3

RUN apk --no-cache add coreutils 

CMD [ "env" ]

docker-compose.yml:

test:
    build: .
    dockerfile: Dockerfile
    environment:
        - HOST_HOSTNAME=${HOSTNAME}

Expected output:

test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=63d376bcaa02
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root

Actual output:

test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=63d376bcaa02
test_1  | HOST_HOSTNAME=
test_1  | HOME=/root
  • host distribution and version ( OSX 10.10.x, OSX 10.11.x, Windows, etc )

OSX:
10.11.4 (15E65)

Docker for Mac:
Version 1.11.1-beta11 (build: 6974)
37559e5f6acd56a4810963acc7001e88f2d88017

docker-compose:
docker-compose version 1.7.1, build 0a9ab35

1 Like
2

Thanks for the report.
Unfortunately I’m unable to reproduce…

$ export HOSTNAME=foo
$ docker compose up
Creating foo_test_1
Attaching to foo_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=125412e7116e
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root
foo_test_1 exited with code 0

The only thing that I can think of is that variable HOSTNAME isn’t exported to the child process, in this case, docker-compose.
For example, the following won’t work…

HOSTNAME=foo
docker-compose up
3

I see. So the issue is that even though the variable is set in my shell, I have to export it? This is confusing, I haven’t seen that kind of behavior before. Eg:

> set | grep HOSTNAME
HOSTNAME=foo
> docker-compose up
WARNING: The HOSTNAME variable is not set. Defaulting to a blank string.
Starting dockercomposeenvvariables_test_1
Attaching to dockercomposeenvvariables_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=2d4d5e216cb6
test_1  | HOST_HOSTNAME=
test_1  | HOME=/root
dockercomposeenvvariables_test_1 exited with code 0
> export HOSTNAME=${HOSTNAME}
> docker-compose up
Recreating dockercomposeenvvariables_test_1
Attaching to dockercomposeenvvariables_test_1
test_1  | PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
test_1  | HOSTNAME=9cb89136063e
test_1  | HOST_HOSTNAME=foo
test_1  | HOME=/root
dockercomposeenvvariables_test_1 exited with code 0

Is this intended behavior? I’ve never had to export variables that were already set in my shell before.

4

That’s pretty standard. Everything that is not EXPORTed will not be passed to other processes. If you’d like to know more:

  • Note that this is not docker-compose related, but really a shell-specific behavior. Indeed, it is the shell that’ responsible for passing or not variables to the sub-process’s environment, and it this case it does not because the variable was not marked for EXPORT.
2 Likes
5

Ok, consider this closed: I’ve learned something new.

6

I saw this error message. In my case, the confusion was caused by the fact that I was using .env to set the environment for Docker compose, and I was running docker-compose commands in a child folder of the folder holding docker-compose.yml and .env.

1 Like
7

Hi, I’m having the same problem, there is something I really don’t get about variable substitution.
I tried the example above and everything is fine if my env variable is “HOSTNAME”, but I’m not able to interpolate new variables in the compose file, for example “FOO”:

# example/Dockerfile
FROM alpine

# example/docker-compose.yml
version: '3'

services:
  test:
    build: .
    environment:
      FOO: $FOO
    command: echo $FOO

# Terminal
$ export FOO=bar
$ sudo docker-compose up
WARNING: The FOO variable is not set. Defaulting to a blank string.
Starting example_test_1 ... 
Starting example_test_1 ... done
Attaching to example_test_1
test_1  | 
example_test_1 exited with code 0
1 Like
8

I figured it out, I’m posting this in case someone might need it.
My problem was that the environment variables are not preserved by default when sodoing.
So, in order for compose variable substitution to work, when the user is not added to docker group, you must add the -E flag to sudo.
For the example above:

$ export FOO=bar
$ sudo -E docker-compose up
Creating network "example_default" with the default driver
Creating example_test_1 ... 
Creating example_test_1 ... done
Attaching to example_test_1
test_1  | bar
example_test_1 exited with code 0

ps. docker-compose config really helped me debug this, it shows in the shell the result of the variables substitutions in docker-compose.yml.

6 Likes
10

I had this problem a few minutes ago, i had to close the terminal and open new one; all is well now.

11

FWIW, I’m having this problem today. I’m using docker-compose version 1.22.0, build f46880fe on Ubuntu 18.04.1.

The env variable in my docker-compose.yml is ${PWD}, and I’ve confirmed the variable exported. (I also quit and restarted the terminal “just in case”.

When I sudo docker-compose run --service-ports meteor I get the following output:

WARNING: The PWD variable is not set. Defaulting to a blank string.
Starting todos_mongo_1 ... done
ERROR: Cannot create container for service meteor: b'create .: volume name is too short, names should be at least two alphanumeric characters'

I did not see any suggestions above that fix this problem. Is it a bug in this release of docker-compose?

12

I am having the following problem:

The relevant bit of the compose file is:

version: "3.7"
services:
  example:
    ...
    environment:
      ...
      HOST_UID: $HOST_UID
      HOST_GID: $HOST_GID
    ....
The following do perform variable substitution:
HOST_UID=1030 HOST_GID=1029 docker-compose -f docker-compose-prod.yml config
The following do not perform variable substitution:
HOST_UID=1030
HOST_GID=1029
docker-compose -f docker-compose-prod.yml config

This is as expected right?

Interestingly, when I change the compose spec to be the following:

version: "3.7"
services:
  example:
    ...
    environment:
      ...
      HOST_UID: $UID
      HOST_GID: $GROUPS
      SOMEOTHER: $SOMEOTHER
      PWD: $PWD
    ....

then the following behaviour is observed in a brand new shell:

$ SOMEOTHER=123
$ echo $SOMEOTHER
123
$ echo $UID
1000
$ echo $GROUPS
1000
$ echo $PWD
/home/.../src/...
$ docker-compose -f docker-compose-prod.yml config

It gives back:

    environment:
      HOST_GID: ''
      HOST_UID: '1000'
      PWD: /home/.../src/...django-backend
      SOME_OTHER: '123'

So here we see that an example of a variable that compose can see is PWD. What is the name for this class of variables? It would be nice if compose could also see all the default variables that are set in a brand new shell. Why is this not the case?

Another interesting observation. The behaviour is odd even when exporting the variables (again in a brand new shell)!

$ SOMEOTHER=123
$ echo $SOMEOTHER
123
$ echo $UID
1000
$ echo $GROUPS
1000
$ echo $PWD
/home/.../src/...
$ # The above is same as before, below we now export these variables
$ export SOMEOTHER
$ export UID
$ export GROUPS
$ # No need to export PWD
$ docker-compose -f docker-compose-prod.yml config

It gives back:

WARNING: The GROUPS variable is not set. Defaulting to a blank string.
...
 environment:
      ...
      HOST_GID: ''
      HOST_UID: '1000'
      PWD: /home/.../src/...
      SOME_OTHER: '123'

Why is the GROUPS variable not set!? Something is fishy here.

Finally:

docker-compose -v
docker-compose version 1.22.0, build f46880fe

docker -v
Docker version 18.06.1-ce, build e68fc7a
13

I use

HOSTNAME=$HOSTNAME docker-compose -f patrickz.yml config

patrickz.yml

version: '3'
services:
  patrickz:
    image: patrickz/ubuntu
    container_name: patrickz
    restart: always
    hostname: $HOSTNAME
    ports:
      - 19999:19999
    environment:
      - TZ=Asia/Bangkok

Note: If the host’s hostname is changed after container start. Container’s hostname will not change.

14

There’s definitely something wrong here… I can get “The FOO variable is not set. Defaulting to a blank string” to go away using sudo -E docker-compose build, but then it breaks when it his my image: name:${tag}
“ERROR: Couldn’t connect to Docker daemon at http+docker://localhost - is it running? If it’s at a non-standard location, specify the URL with the DOCKER_HOST environment variable.”

If I add the tag manually into .env instead of exporting it, then it does not break. No idea what is going on, so I’ll have to just manually add it to .env for now.

15

Wow, that was a lifesaver. Thanks @bertoldi! sudo -E docker-compose up worked like a charm.

16

I ran into this problem yesterday (12/31/2019). In my case, it was caused by having used an alternative install option: “Install as a container”. docker-compose was being run from a container and that container didn’t have access to my host’s environment variables. Maybe it’s a bug, maybe it’s a feature. I suppose the script could be updated to somehow send all host environment variables to the container.

In any case, I solved the problem by reinstalling docker-compose, using a more conventional option that installed the executable into my path. Works fine now.

17

Awesome! Thanks for this!

18

@bertoldi Thanks you so much.

19

@bertoldi thank you, man! You are incredible, saved me days of work:)

20

Hi,
I have the same problem:
after the command: docker-compose up -d

The system do not reconise my machine.

ERROR: Couldn’t connect to Docker daemon at http+docker://localhost - is it running?

If it’s at a non-standard location, specify the URL with the DOCKER_HOST environment variable.

command: echo $HOSTNAME
gives: puyuelo-W65-67SZ which my machine name.
if I had in docker-compose.yml

environment:
        - HOST_HOSTNAME=$(puyuelo-W65-67SZ}
is it enough to work ?
21

Thanks a lot for that!