Docker Community Forums

Share and learn in the Docker community.

[SOLVED] Unable to pass (environment) variables (or secrets) into docker stack deploy (mysql)

Hi all,

I’m new to the forum and docker swarm; relatively new to docker in general. I’ve been struggling with an seemingly simple set-up for several days now. There is probably a rookie mistake hidden in here, but I can’t get my head around it and I’m looking forward to your help!

Issue: I can’t get docker stack deploy to work with the environment runtime argument. I can get the configuration to work with docker-compose.

The configuration that I’m sharing is based on MySQL docker documentation. From what I can tell so far, the issue is not related to MySQL.

Contents of docker-compose.yml:

version: '3.8'

services:
  db:
    image: mysql:latest
    command: --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD=123

  adminer:
    image: adminer
    ports:
      - 8080:8080

networks:
  default:
    name: db_network
    driver: overlay

Actual behavior:

$ sudo docker stack deploy -c docker-compose.yml mysql-tester
Creating network db_network
Creating service mysql-tester_adminer
Creating service mysql-tester_db
$ sudo docker stack ps mysql-tester
ID             NAME                     IMAGE            NODE      DESIRED STATE   CURRENT STATE            ERROR                       PORTS
ux3wysxllr3e   mysql-tester_adminer.1   adminer:latest   NAS     Running         Running 55 seconds ago
nd165ye84yfs   mysql-tester_db.1        mysql:latest     NAS     Running         Starting 1 second ago
pzko4nzra8fk    \_ mysql-tester_db.1    mysql:latest     NAS     Shutdown        Failed 6 seconds ago     "task: non-zero exit (1)"
yyxwor45t6jk    \_ mysql-tester_db.1    mysql:latest     NAS     Shutdown        Failed 18 seconds ago    "task: non-zero exit (1)"
izddw0mvn0e9    \_ mysql-tester_db.1    mysql:latest     NAS     Shutdown        Failed 30 seconds ago    "task: non-zero exit (1)"
u21trxihjmas    \_ mysql-tester_db.1    mysql:latest     NAS     Shutdown        Failed 42 seconds ago    "task: non-zero exit (1)"
$ # the following two command only work if a new container is started and not yet crashed -- in practice I only get output once in every few tries of the same command
$ sudo docker logs $(sudo docker ps -q --filter name=mysql-tester_db)
2021-08-25 19:35:26+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2021-08-25 19:35:28+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-08-25 19:35:28+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2021-08-25 19:35:28+00:00 [ERROR] [Entrypoint]: Database is uninitialized and password option is not specified
    You need to specify one of the following:
    - MYSQL_ROOT_PASSWORD
    - MYSQL_ALLOW_EMPTY_PASSWORD
    - MYSQL_RANDOM_ROOT_PASSWORD

From docker inspect it appears that the Config.Cmd reflects my docker-compose.yml settings, but the Config.Env does not.

docker inspect
$ sudo docker inspect $(sudo docker ps -q --filter name=mysql-tester_db)
[
    {
        "Id": "85d6b382eb13529439484a259cbb69dddcb5835943564d263549bfbd124ae164",
        "Created": "2021-08-25T19:36:30.67405673Z",
        "Path": "docker-entrypoint.sh",
        "Args": [
            "--default-authentication-plugin=mysql_native_password"
        ],
        "State": {
            "Status": "running",
            "Running": true,
            "Paused": false,
            "Restarting": false,
            "OOMKilled": false,
            "Dead": false,
            "Pid": 9224,
            "ExitCode": 0,
            "Error": "",
            "StartedAt": "2021-08-25T19:36:34.912297707Z",
            "FinishedAt": "0001-01-01T00:00:00Z",
            "StartedTs": 1629920194,
            "FinishedTs": -62135596800
        },
        "Image": "sha256:5a4e492065c722ec8cc7413552bafc6fd5434c5ad90797e898ccc4e347e21aa5",
        "ResolvConfPath": "/volume1/@docker/containers/85d6b382eb13529439484a259cbb69dddcb5835943564d263549bfbd124ae164/resolv.conf",
        "HostnamePath": "/volume1/@docker/containers/85d6b382eb13529439484a259cbb69dddcb5835943564d263549bfbd124ae164/hostname",
        "HostsPath": "/volume1/@docker/containers/85d6b382eb13529439484a259cbb69dddcb5835943564d263549bfbd124ae164/hosts",
        "LogPath": "/volume1/@docker/containers/85d6b382eb13529439484a259cbb69dddcb5835943564d263549bfbd124ae164/log.db",
        "Name": "/mysql-tester_db.1.5lw73wrxzwppkhy26ym0vk9vh",
        "RestartCount": 0,
        "Driver": "btrfs",
        "Platform": "linux",
        "MountLabel": "",
        "ProcessLabel": "",
        "AppArmorProfile": "docker-default",
        "ExecIDs": null,
        "HostConfig": {
            "Binds": null,
            "ContainerIDFile": "",
            "LogConfig": {
                "Type": "db",
                "Config": {}
            },
            "NetworkMode": "default",
            "PortBindings": {},
            "RestartPolicy": {
                "Name": "",
                "MaximumRetryCount": 0
            },
            "AutoRemove": false,
            "VolumeDriver": "",
            "VolumesFrom": null,
            "CapAdd": null,
            "CapDrop": null,
            "CgroupnsMode": "host",
            "Dns": null,
            "DnsOptions": null,
            "DnsSearch": null,
            "ExtraHosts": null,
            "GroupAdd": null,
            "IpcMode": "private",
            "Cgroup": "",
            "Links": null,
            "OomScoreAdj": 0,
            "PidMode": "",
            "Privileged": false,
            "PublishAllPorts": false,
            "ReadonlyRootfs": false,
            "SecurityOpt": null,
            "UTSMode": "",
            "UsernsMode": "",
            "ShmSize": 67108864,
            "Runtime": "runc",
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=8.0",
                "MYSQL_VERSION=8.0.26-1debian10"
            ],
            "ConsoleSize": [
                0,
                0
            ],
            "Isolation": "default",
            "CpuShares": 0,
            "Memory": 0,
            "NanoCpus": 0,
            "CgroupParent": "",
            "BlkioWeight": 0,
            "BlkioWeightDevice": null,
            "BlkioDeviceReadBps": null,
            "BlkioDeviceWriteBps": null,
            "BlkioDeviceReadIOps": null,
            "BlkioDeviceWriteIOps": null,
            "CpuPeriod": 0,
            "CpuQuota": 0,
            "CpuRealtimePeriod": 0,
            "CpuRealtimeRuntime": 0,
            "CpusetCpus": "",
            "CpusetMems": "",
            "Devices": null,
            "DeviceCgroupRules": null,
            "DeviceRequests": null,
            "KernelMemory": 0,
            "KernelMemoryTCP": 0,
            "MemoryReservation": 0,
            "MemorySwap": 0,
            "MemorySwappiness": null,
            "OomKillDisable": false,
            "PidsLimit": null,
            "Ulimits": [],
            "CpuCount": 0,
            "CpuPercent": 0,
            "IOMaximumIOps": 0,
            "IOMaximumBandwidth": 0,
            "MaskedPaths": [
                "/proc/asound",
                "/proc/acpi",
                "/proc/kcore",
                "/proc/keys",
                "/proc/latency_stats",
                "/proc/timer_list",
                "/proc/timer_stats",
                "/proc/sched_debug",
                "/proc/scsi",
                "/sys/firmware"
            ],
            "ReadonlyPaths": [
                "/proc/bus",
                "/proc/fs",
                "/proc/irq",
                "/proc/sys",
                "/proc/sysrq-trigger"
            ]
        },
        "GraphDriver": {
            "Data": null,
            "Name": "btrfs"
        },
        "Mounts": [
            {
                "Type": "volume",
                "Name": "05a451cca22a94dfcffc5b73690f4803ae470e60bf97e7ff45bc9ef19e08b154",
                "Source": "/volume1/@docker/volumes/05a451cca22a94dfcffc5b73690f4803ae470e60bf97e7ff45bc9ef19e08b154/_data",
                "Destination": "/var/lib/mysql",
                "Driver": "local",
                "Mode": "",
                "RW": true,
                "Propagation": ""
            }
        ],
        "Config": {
            "Hostname": "85d6b382eb13",
            "Domainname": "",
            "User": "",
            "AttachStdin": false,
            "AttachStdout": false,
            "AttachStderr": false,
            "ExposedPorts": {
                "3306/tcp": {},
                "33060/tcp": {}
            },
            "Tty": false,
            "OpenStdin": false,
            "StdinOnce": false,
            "Env": [
                "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
                "GOSU_VERSION=1.12",
                "MYSQL_MAJOR=8.0",
                "MYSQL_VERSION=8.0.26-1debian10"
            ],
            "Cmd": [
                "--default-authentication-plugin=mysql_native_password"
            ],
            "Image": "mysql:latest",
            "Volumes": {
                "/var/lib/mysql": {}
            },
            "WorkingDir": "",
            "Entrypoint": [
                "docker-entrypoint.sh"
            ],
            "OnBuild": null,
            "Labels": {
                "com.docker.stack.namespace": "mysql-tester",
                "com.docker.swarm.node.id": "roe85itnvzbdesc4y1kc7387d",
                "com.docker.swarm.service.id": "sc5yg9m0b0bwgrxm7qwbiy47p",
                "com.docker.swarm.service.name": "mysql-tester_db",
                "com.docker.swarm.task": "",
                "com.docker.swarm.task.id": "5lw73wrxzwppkhy26ym0vk9vh",
                "com.docker.swarm.task.name": "mysql-tester_db.1.5lw73wrxzwppkhy26ym0vk9vh"
            },
            "DDSM": false
        },
        "NetworkSettings": {
            "Bridge": "",
            "SandboxID": "18219b8b9ed4d07b19585c80f19c6950a9cacaddbad6fb8ba3f542d4a9c22572",
            "HairpinMode": false,
            "LinkLocalIPv6Address": "",
            "LinkLocalIPv6PrefixLen": 0,
            "Ports": {
                "3306/tcp": null,
                "33060/tcp": null
            },
            "SandboxKey": "/var/run/docker/netns/18219b8b9ed4",
            "SecondaryIPAddresses": null,
            "SecondaryIPv6Addresses": null,
            "EndpointID": "",
            "Gateway": "",
            "GlobalIPv6Address": "",
            "GlobalIPv6PrefixLen": 0,
            "IPAddress": "",
            "IPPrefixLen": 0,
            "IPv6Gateway": "",
            "MacAddress": "",
            "Networks": {
                "db_network": {
                    "IPAMConfig": {
                        "IPv4Address": "10.0.39.38"
                    },
                    "Links": null,
                    "Aliases": [
                        "85d6b382eb13"
                    ],
                    "NetworkID": "0109v22egwpfwi6dn3sguu1wc",
                    "EndpointID": "bfcfc9ece87d65a31191e926701a2da1fc747c0faeab869cebee69a60b957e92",
                    "Gateway": "",
                    "IPAddress": "10.0.39.38",
                    "IPPrefixLen": 24,
                    "IPv6Gateway": "",
                    "GlobalIPv6Address": "",
                    "GlobalIPv6PrefixLen": 0,
                    "MacAddress": "02:42:0a:00:27:26",
                    "DriverOpts": null
                }
            }
        }
    }
]

Expected behavior would be that the environmental variable MYSQL_ROOT_PASSWORD is set as defined in the container.

When trying the same thing with docker-compose, everything works fine.

Expected behavior (docker compose)
$ docker-compose up -d
WARNING: The Docker Engine you're using is running in swarm mode.

Compose does not use swarm mode to deploy services to multiple nodes in a swarm. All containers will be scheduled on the current node.

To deploy your application across the swarm, use `docker stack deploy`.

Creating network "db_network" with driver "overlay"
Creating docker_adminer_1 ... done
Creating docker_db_1      ... done
$ # wait for several minutes
$ docker logs docker_db_1
2021-08-25 19:25:29+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2021-08-25 19:25:31+00:00 [Note] [Entrypoint]: Switching to dedicated user 'mysql'
2021-08-25 19:25:31+00:00 [Note] [Entrypoint]: Entrypoint script for MySQL Server 8.0.26-1debian10 started.
2021-08-25 19:25:31+00:00 [Note] [Entrypoint]: Initializing database files
2021-08-25T19:25:31.984826Z 0 [System] [MY-013169] [Server] /usr/sbin/mysqld (mysqld 8.0.26) initializing of server in progress as process 43
2021-08-25T19:25:32.049285Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-08-25T19:25:45.424038Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-08-25T19:26:15.809887Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-08-25T19:26:15.811557Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-08-25T19:26:15.997156Z 6 [Warning] [MY-010453] [Server] root@localhost is created with an empty password ! Please consider switching off the --initialize-insecure option.
2021-08-25 19:27:27+00:00 [Note] [Entrypoint]: Database files initialized
2021-08-25 19:27:27+00:00 [Note] [Entrypoint]: Starting temporary server
2021-08-25T19:27:28.258888Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.26) starting as process 90
2021-08-25T19:27:28.647446Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-08-25T19:27:31.484457Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-08-25T19:27:35.195883Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-08-25T19:27:35.196124Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-08-25T19:27:35.261837Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-08-25T19:27:35.262681Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-08-25T19:27:35.313082Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2021-08-25T19:27:35.390931Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Socket: /var/run/mysqld/mysqlx.sock
2021-08-25T19:27:35.391814Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26'  socket: '/var/run/mysqld/mysqld.sock'  port: 0  MySQL Community Server - GPL.
2021-08-25 19:27:35+00:00 [Note] [Entrypoint]: Temporary server started.
Warning: Unable to load '/usr/share/zoneinfo/iso3166.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/leap-seconds.list' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone.tab' as time zone. Skipping it.
Warning: Unable to load '/usr/share/zoneinfo/zone1970.tab' as time zone. Skipping it.

2021-08-25 19:27:48+00:00 [Note] [Entrypoint]: Stopping temporary server
2021-08-25T19:27:48.723999Z 10 [System] [MY-013172] [Server] Received SHUTDOWN from user root. Shutting down mysqld (Version: 8.0.26).
2021-08-25T19:28:13.893964Z 0 [System] [MY-010910] [Server] /usr/sbin/mysqld: Shutdown complete (mysqld 8.0.26)  MySQL Community Server - GPL.
2021-08-25 19:28:14+00:00 [Note] [Entrypoint]: Temporary server stopped

2021-08-25 19:28:14+00:00 [Note] [Entrypoint]: MySQL init process done. Ready for start up.

2021-08-25T19:28:15.316674Z 0 [System] [MY-010116] [Server] /usr/sbin/mysqld (mysqld 8.0.26) starting as process 1
2021-08-25T19:28:15.414013Z 1 [System] [MY-013576] [InnoDB] InnoDB initialization has started.
2021-08-25T19:28:17.930795Z 1 [System] [MY-013577] [InnoDB] InnoDB initialization has ended.
2021-08-25T19:28:20.316099Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1 is enabled for channel mysql_main
2021-08-25T19:28:20.316591Z 0 [Warning] [MY-013746] [Server] A deprecated TLS version TLSv1.1 is enabled for channel mysql_main
2021-08-25T19:28:20.496108Z 0 [Warning] [MY-010068] [Server] CA certificate ca.pem is self signed.
2021-08-25T19:28:20.496659Z 0 [System] [MY-013602] [Server] Channel mysql_main configured to support TLS. Encrypted connections are now supported for this channel.
2021-08-25T19:28:20.616333Z 0 [Warning] [MY-011810] [Server] Insecure configuration for --pid-file: Location '/var/run/mysqld' in the path is accessible to all OS users. Consider choosing a different directory.
2021-08-25T19:28:20.811543Z 0 [System] [MY-011323] [Server] X Plugin ready for connections. Bind-address: '::' port: 33060, socket: /var/run/mysqld/mysqlx.sock
2021-08-25T19:28:20.818458Z 0 [System] [MY-010931] [Server] /usr/sbin/mysqld: ready for connections. Version: '8.0.26'  socket: '/var/run/mysqld/mysqld.sock'  port: 3306  MySQL Community Server - GPL.

I’ve tried the same configuration with docker secrets, but that leads to the same observations.

Configuration with docker secrets

Setting secret:

$ echo MYSQL_ROOT_PASSWORD=123 >> ./mysql_root_password.txt
$ cat mysql_root_password.txt
MYSQL_ROOT_PASSWORD=123

Updating docker-compose.yml:

version: '3.8'

secrets:
  MYSQL_ROOT_PASSWORD:
    file: ./mysql_root_password.txt

services:
  db:
    image: mysql:latest
    command: --default-authentication-plugin=mysql_native_password
    environment:
      - MYSQL_ROOT_PASSWORD_FILE=/run/secrets/MYSQL_ROOT_PASSWORD
    secrets:
      - MYSQL_ROOT_PASSWORD

  adminer:
    image: adminer
    ports:
      - 8080:8080

networks:
  default:
    name: db_network
    driver: overlay

With this setup, the results are exactly the same.

Versions/builds (Synology NAS, linux based):

$ sudo docker-compose version
docker-compose version 1.28.5, build 324b023a
docker-py version: 4.4.4
CPython version: 3.7.10
OpenSSL version: OpenSSL 1.1.0l  10 Sep 2019
$ sudo docker version
Client:
 Version:           20.10.3
 API version:       1.41
 Go version:        go1.15.6
 Git commit:        b455053
 Built:             Mon Jun 21 02:03:40 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.3
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.15.6
  Git commit:       a3bc36f
  Built:            Mon Jun 21 02:05:07 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.4.3
  GitCommit:        ea3508454ff2268c32720eb4d2fc9816d6f75f88
 runc:
  Version:          v1.0.0-rc93
  GitCommit:        31cc25f16f5eba4d0f53e35374532873744f4b31
 docker-init:
  Version:          0.19.0
  GitCommit:        ed96d00

I have searched the web quite extensively, but perhaps I’m searching with the wrong keywords (usually end up in issues that involve passing environment variables from host to docker, e.g. VAR=$(VAR)). I’ll post some of my keywords here so that others may find it: docker stack swarm environment variable ignored etc.

Help is very much appreciated!

1 Like

Dear community members,

I’m still struggling with this issue and I’d be grateful for your advice. I’m happy to test suggestions and provide results / debug info / …
Please don’t be shy!

Best regards,

I am afraid you will not like the answer…

Swarm mode always was and still is broken on Synogy’s docker distribution. This is one of the cases where the distributor modified it’s own Docker package to deviate in behavior.

If you need the Swarm mode, you can setup a VMM machine with Debian or Ubuntu and run Docker there.

Hi @meyay,

Well I can’t say I like the answer, but I am very grateful to get some key information that at least explains why I run into trouble (which in turn provides me a slight sense of peacefulness). I thought of Docker as the ultimate OS independent solution, so it is valuable to know that there are exceptions to that rule – regardless of the reason why.

A VMM machine to run docker feels like an additional complication that I would like to avoid. Perhaps you could share your thoughts on two alternative thoughts:

  • -Would the issue be resolved if Docker was installed from a decent distribution (e.g. not from Synology’s software center)? Probably not, as that would be easier than your suggestion.

  • -The reason why I looked into Docker swarm is two-fold. Perhaps there is a fitting non-swarm solution (particularly solving my first issue) that I am not familiar with?

    • 1.First and foremost, I wanted to maintain multiple collections of services and their configuration (e.g. compose files) and have them work nicely in parallel. Docker-compose usually throws warnings if you have multiple of such files running next to each other (possible orphans). Also, it is hard to maintain a structured overview of all the groups that are running.

    • 2.Second, I was interested in docker secrets.

If not, I’ll look further into the VMM possibilities (or just resort to a clutter of services through docker-compose).

It is neither possible to use a package from any other os distribution, nor is it possible to replace the docker binaries with those from docker’s official binary release package.

Even though Synology claims it’s a third party product - which is true for the docker sources they forked their code base from, but not so much after they heavily modified behavior according their needs.

Containers typicaly become orphaned if multipe compose files are stored in the same folder or the same compose file is modified and used to deploy “separate” compose stacks. Docker compose identifies the containers of a compose stack by the container label com.docker.compose.project. This is why it’s important to use unique --project-name settings per compose stack. If no project name is specified the folder name is used as project name.

When it commes to secrets: afair it is possible to use a compose file based on the v3.x schema and use secrets with docker compose as well (I always felt that this is a bug).

VMM realy is the cleanest solution: install the docker-ce package from docker’s offical repositories and run Portainer to manage your containers and you get vanilla Docker behavior and a management UI that does not opinionate what features to hide from the user. VMM itself is a nice ui arround KVM/QEMU - which actualy is an excellent hypervisor and allows paravirtualization for network and storage.

Thank you for elaborating. I was not familiar with the --project-name option. It sounds as if that would solve my first problem. Nevertheless, I’ll have a dive with VMM + Portainer as an exercise.

Thank you once again for pointing out the heart of the issue and possible workarounds.