Build-time Secrets

For open-source projects (or those with only open-source dependencies), most teams never have to deal with this.

But for projects with private dependencies (most companies I know of), which require additional api or SSH keys, how are people getting them? Build-time arguments were added in Docker 1.9, but the docs explicitly state to not use them for secret passing.

There are only two ways I know of to get a build-time secret into a container.

  1. Don’t do it at build time. We call this a two-phase build. First phase builds a container with all the OS dependencies and anything else that may be openly available. Now, in the second phase, run the result of the build, mounting a volume of secrets (so they won’t be persisted). On startup of this artifact, detect that secrets are available, do the secondary, private dependency retrievals, and EXIT. Commit the result. Requires host and container knowledge of this process :frowning:
  2. Super hacky, pass secret contents over docker bridge network during build using netcat. Requires host and container knowledge of this process :frowning:

Does anyone else have suggestions?

1 Like

It’s an outstanding known issue. My recommendation is that if you can do whatever git clone type operations that need SSH access outside of the container, then COPY them in, you should do that in the interim.

Thanks for reply! Agreed, we can do that. Though, moving the problem out of the container kind of negates reason for using a container in the first place. Especially when you get to private dependencies with python (pip), ruby (gems), php (composer), those are versioned languages and tools that would need to exist outside the container to make this a reality.

those are versioned languages and tools that would need to exist outside the container to make this a reality.

Yes, I agree – Depending on how complex your needs are, I think it’d be a reasonable approach to make a “build pipeline” where you do a couple of docker run --rm commands with ~/.ssh and $PWD bind mounted, and do the npm install / git clone type commands in there, then have a final step which docker builds the finished image using COPY as mentioned. That would allow you to still take advantage of the library images and isolation of containers while keeping secrets out of the final image.

This can all be automated using make / bash / $YOUR_BUILD_TOOL.

It’d be awesome to have a more detailed reason as to why build-time variables are inappropriate for secret-passing. @nathanleclaire ? :slight_smile:

For reference, I’ve been using a 2-step process for this sort of thing - basically two Dockerfiles - the first named Dockerfile.build, where the secrets are copied in (in my case, a .npmrc file) at build time. Then a later instruction simply deletes it. Obviously this is no good on its own, because the .npmrc file still exists in an earlier layer. Finally the whole filesystem is tar.gzed and the CMD instruction is set to cat it.

After this the original “build” image is docker run and piped to a file, then finally the “real” Dockerfile is built, with an ADD instruction to unroll the tarball into the new image.

The approach works, and as long as you’re sure to delete the secrets before tarballing the filesystem, your secrets are safe. But it’s brittle and a little gross.

I’ve been toying with using build args for this (notwithstanding the explicit recommendation in the docs to not do this). A sample Dockerfile could look something like:

FROM node:6

COPY package.json .
ARG NPMRC
RUN echo "$NPMRC" > .npmrc \
  && npm install \
  && rm .npmrc
ENV NPMRC unset

# ...

Then to build:

$ docker build --build-arg NPMRC="$(cat .npmrc)" -t foo .

This approach works, the .npmrc file doesn’t persist in the image, and the value of the NPMRC build arg is only available for the RUN ... npm install instruction. So in theory this is safe, but again, the docs say this is a bad idea (but then don’t explain why).

Ok, to answer my own question:

It’d be awesome to have a more detailed reason as to why build-time variables are inappropriate for secret-passing.

Looks like there’s a really good reason for this: build-time args persist in the image (!) as metadata, just not as environment variables at runtime.

To wit:

$ docker inspect -f '{{.ContainerConfig.Cmd}}' the-image-name
[|1 NPMRC=the_contents_of_the_original_value /bin/sh -c echo "$NPMRC" > .npmrc   && npm install   && rm .npmrc]

So essentially, using build-time args for secrets really is a bad idea!

4 Likes

I wrote https://github.com/abourget/secrets-bridge to address the build-time secrets problem.

It creates a throw-away configuration that you can pass as arguments, during the build process, it will connect to the host and fetch the secrets, use them, and then you can kill the host bridge. Even if the build-args are saved somewhere, they become useless the moment the server is killed.

The server supports SSH Agent Forwarding, tunnelled through a TLS websocket communication. It works on Windows too !

Hope this helps.