.dockerignore in multi-stage builds

Archive Feature Requests
1

Hi,

my task is to build the Single-Page-Application (Stage 1) and then serve it with NGNIX (Stage 2)

FROM node:8.11.4-alpine AS build
WORKDIR /usr/local/app
COPY . .
RUN npm install
RUN npm run build

FROM nginx:1.15.2-alpine
COPY nginx/nginx.conf /etc/nginx/nginx.conf
COPY nginx/conf.d/default.conf /etc/nginx/conf.d/
COPY --from=build /usr/local/app/dist/MY-APP /usr/share/nginx/html

In Stage 1 I copy all the src files into the build container. To improve this process I created a .dockerignore (as it is suggested in the docs). For the build I obviously do not need the nginx files. Therefore they are part of the .dockerignore.

dist
e2e
nginx
node_modules
.editorconfig
.gitignore
npm-debug.log
README.md
tslint.json

The problem - as soon as the Stage 2 begins, the ngnix entry in the .dockerignore file prevents that the ngnix files can be copied! So the only working solution so far is to remove ngnix from the .dockerignore and therefore copy it into without any need the build container (Stage 1). This is not how it should be!

Feature request - provide a better multi-stage .dockerignore support.
There are probably several solution designs possible.
My personal solution idea would be a ā€œblock definitionā€ in .dockerignore similar to the one used in the Dockerfile (here it is separated by the ā€˜FROMā€™ keyword)

So, what is your opinion to this guys?
Any other solutions or thoughts?

5 Likes
2

I totally agree. For me this currently makes multi stage builds useless. Surprised this hasnā€™t caught more attention. I mean in the typical setup you have one or more builder stages and one runner stage and they typically need quite different .dockerignore contexts to make them snappy and and cacheable. Or am I missing something?

3

+1, I have some larger test files that I donā€™t want in the build context unless Iā€™m targeting the test stage

It would be OK if I had to have sections of the .dockerignore match the names or orders of stages in the dockerfile.

4

A similar issue here, have 2 jobs, one uses Dockerfile.One and the other uses Dockerfile.Two. Now for e.g out of A & B & C files I need the A & B added through the 1st job & B & C through the 2nd job. The issue is that C is a huge doc and is included in the .dockerignore and wonā€™t be used when building the B & C case.

How can I tell it to ignore the .dockerignore rule for only this specific time?

5

A multi-stage build is still a single build. As such, it begins by copying your entire build context directory (minus what is excluded via .dockerignore) from the client where you invoke the build command, to the server where the build actually happens.

This does not mean that every file in the build context directory gets copied to every build stage. In your example, your nginx files will not affect the first stage at all.

The purpose of the .dockerignore file is not to make the build containers or the resulting image ā€œsnappierā€. It is just to prevent files from being copied to the server in the first step.

7

@rajchaudhuri ,

The point here is that the .dockerignore affects MORE than just the first stage of the Dockerfile. In fact, it applies to EVERY COPY in EVERY stage of the build. So, a multi-stage Dockerfile to build a node.js application needs a .dockerignore like this:

node_modules

This is important so that the node_modules arenā€™t in the build context ā€“ which makes is HUGE.

Once the app has been built, a later stage in the Dockerfile creates the container image for the app. This stage MUST copy node_modules from the build stage. However, the .dockerignore file will prevent this.

!!! THIS IS UNACCEPTABLE !!!

@lotosotol,

Sadly, I donā€™t think that this is possibleā€¦ :frowning:

8

@rajchaudhuri already explained how it is actually implemented. Everyone else: please read the documentation about how .dockerignore works and what it actually affects:

https://docs.docker.com/build/building/context/#dockerignore-files

Only COPY instructions (without the --from argument) are affected from entries in .dockerignore, as they are used to copying files from the build context into the image. The .dockerignore file makes sure ignored files are not transferred from the local directory to the build context.

This is what the docs say about copying from stages of a multi-staged build:

Optionally COPY accepts a flag --from=<name> that can be used to set the source location to a previous build stage (created with FROM .. AS <name> ) that will be used instead of a build context sent by the user. In case a build stage with a specified name canā€™t be found an image with the same name is attempted to be used instead.

9

Since I found this topic again, I add some notes and share a promising solution.

So if we are talking about copying files from the host (not from other stages) to different stages, I had problem with that before, but these are usually solvable by changing the folder structure. The source code of the different stages could be in separate folders similarly to how we could have multiple Docker projects in subfolders. This works of course when each stage has different files and no folder needed to be copied to multiple stages. If that is needed, we could always copy those files from other stages instead of from the host.

So I think the requirement to have different dockerignore files for different stages comes from a project structure that we donā€™t want to or canā€™t change for some reason. For example because an IDE understands that structure and we donā€™t know how we should handle a different structure in that IDE.

We could ask the question whether IDEs should be improved or Docker should have a feature to have multiple dockerignore files.

It seems that the developers listened to the users as there is a feature in buildkit that supports excluding files in the COPY instruction which is almost like having multiple dockerignore files, but better because it can be different in each COPY.

Related part copied from the documentation

COPY [--exclude=<path> ...] <src> ... <dest>

The --exclude flag lets you specify a path expression for files to be excluded.

The path expression follows the same format as <src>, supporting wildcards and matching using Goā€™s filepath.Match rules. For example, to add all files starting with ā€œhomā€, excluding files with a .txt extension:

COPY --exclude=*.txt hom* /mydir/

You can specify the --exclude option multiple times for a COPY instruction. Multiple --excludes are files matching its patterns not to be copied, even if the files paths match the pattern specified in <src>. To add all files starting with ā€œhomā€, excluding files with either .txt or .md extensions:

COPY --exclude=*.txt --exclude=*.md hom* /mydir/

It is still not a stable feature so the documentation also says:

Note

Not yet available in stable syntax, use docker/dockerfile:1.7-labs version.

That means, you have to add the following to the first line of the Dockerfile

# syntax=docker/dockerfile:1.7-labs
2 Likes
10

Oh thanks for the tip !

I was searching for such feature a few days ago. Copying everything except a given file I need to copy a few instructions later (in an another stage)

1 Like