Secrets in clear-text on Swarm Host

Docker Engine Swarm
1

Hello all,

we recently started to employ Swarm Secrets to manage passwords required by containers. The Secrets documentation emphasizes that all secrets are stored in encrypted form and that secrets are mounted on an in-memory file system inside the container and never persisted to disk.

https://docs.docker.com/engine/swarm/secrets/:

When you deploy, Docker mounts a file under /run/secrets/<secret_name> in the services. These files are never persisted in disk, but are managed in memory.

However, we found that passwords are still present in clear text on Swarm hosts under /var/lib/docker/containers/<id>/mounts/secrets/. Which appears to contradict the statement quoted above. Is this to be expected or did we screw up our setup somehow?

Thanks in advance!
Ralf

$ docker version
Client:
 Version:           18.09.0
 API version:       1.39
 Go version:        go1.10.4
 Git commit:        4d60db4
 Built:             Wed Nov  7 00:48:22 2018
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          18.09.0
  API version:      1.39 (minimum version 1.12)
  Go version:       go1.10.4
  Git commit:       4d60db4
  Built:            Wed Nov  7 00:19:08 2018
  OS/Arch:          linux/amd64
  Experimental:     false
2

I am not sure if this realy is a contradicion, as secrets work like this:

  1. secrets are encypted during network transport from managers to other managers and works.

  2. secrets are encrypted inside the logs of the Raft consensus algorithm (== stored in encrypted form)

  3. only containers having the secret assigned can access this secret from the raft-protocoll and mount it in-memory

  4. after container shutdown the in-memory mount is deleted and flushed

When a container task stops running, the decrypted secrets shared to it are unmounted from the in-memory filesystem for that container and flushed from the node’s memory.

  1. What you see in /var/lib/docker/containers//mounts/secrets/ are the in-memory mounts. Check output of mount | grep /var/lib/docker/containers//mounts/secrets/ to see details. You will see it is neither saved directly on the harddisk nor inside the container.

Though, beeing able to access secrects in plain text from the in-memory mounts in the filesystem , when the container is running seems kind of wrong.

3

@meyay, thanks for your reply.

And that’s the part that surprised me. As you summarized, the documentation for Docker Secrets is all about encrypted storage and transfer of secrets. And at least to me it implies that the requested secrets are private to a container (process).

This is the real bummer. I wonder if this is by design and deemed “OK” or if we somehow messed up our configuration. Or is it even a bug?
If it is by design then I think it should at least be mentioned in the documentation for Secrets.

4

I can verify the behaviour that you are stating. However if you do a df . on the secrets mount you’d get an output like this

Filesystem     1K-blocks  Used Available Use% Mounted on
tmpfs            1990172     8   1990164   1% /var/lib/docker/containers/cdf661b09e03bc8359c93820176af68601c9d69199648272465ec89cfcc7ec4b/mounts/secrets

Note the filesystem is tmpfs so that’s in memory and disappears when the node terminates. In addition that path is only accessible via root.