Hello all! I’ve recently spent some time learning Docker and have been attempting to write my own container images. I have a few questions that I would appreciate to get some thoughts on and learn more about.
Please forgive me if some aspects might be obvious, I’m really new to it but I am very keen to learn!
My interest in Docker containers at the moment lies more on the fact that when I work on a project with someone else, the development environment is the same and introducing new developers to a project can become slightly easier since they wouldn’t have to install any dependencies or run a number of commands they maybe do not understand what they’re for. After reading some of the documentation and watching some videos on how to build a Docker image I started trying to implement it in one of my projects which requires Node to work.
It does work fine (can be accessed here) but there are some aspects I do not fully understand. For instance, if I describe an npm install in a Dockerfile it runs it when building the image. However, does that mean that the node_modules folder is stored inside the image?
I assumed that it does since when the image is built the npm install command seems to run at that point, is this correct? The reason I’m not sure what really happens is that I ended up having to create a volume for node_modules in order to make use of the dependencies, however, I’m not quite sure why, the folder existed in the container, but it was empty.
I’m leaving my files below for anyone to give feedback, please feel free to access the project. I added these files to if you would like to see it in a more contextualized manner. Is there a better way to create an image for a project that needs node dependencies? Here’s the files:
Dockerfile:
# This image makes use of a Node image running on Linux Alpine
FROM node:15-alpine3.13
# A work directory is required to be used by npm install
WORKDIR /var/projects/node-app
# Copy all project files to the container
# Files in the location of this file are copied to WORKDIR in the container
COPY . .
# Makes sure npm is up to date otherwise install dependencies attempts will fail
RUN npm install -g npm
# Install dependencies
RUN npm install
# The process this container should run
CMD ["npm", "start"]
Initial docker-compose (made use of a .env for path but this aspect is not relevant I assume)
version: "3"
services:
proton:
build: ./ # The path to node-app Dockerfile
volumes:
# Allows changes made to project directory to be accessed by the container and persist
- ${PROJECT_PATH}:/var/projects/node-app
The final docker-compose file (this works but why is a Volume necessary for node_modules here?)
version: "3"
services:
proton:
build: ./ # The path to node-app Dockerfile
volumes:
# Allows changes made to project directory to be accessed by the container and persist
- .:/var/projects/node-app
# A volume dedicated to node modules.
- /var/projects/node-app/node_modules
Thank you so much for taking the time to read this!
UPDATE: Striked through content were the links to the project on Github. This was flagged as spam so I have remove them. sorry for the inconvenience.
Your Dockerfile is under the protonmail-theme/docker folder move it to the same folder as your docker-compose file. NPM isn’t installing for you because the WORKDIR you are switching to doesn’t exist. Docker doesn’t create one for you and fails silently.
FROM node:15-alpine3.13
COPY . /var/projects/protonmail-themes
WORKDIR /var/projects/protonmail-themes
RUN npm install -g npm && \
npm install
CMD ["npm", "start"]
Also you are copying all your project files including the docker and docker-compose files to the container which is wrong. You should only copy the files necessary to run the application to the container.
Hi David, thank you so much for taking the time to share your thoughts. I assume you had a look at the repository which has since been updated and now has a slightly different setup.
The context of the file is set in docker-compose.yml and it is currently working as expected, building the same way as if node.dockerfile was in the same folder of the docker-compose.yml file, like so:
version: "3"
services:
proton:
# Building a custom image described in a docker file.
build:
# Setting a context and dockerfile paths allows for Dockerfiles to be stored away in a sub-directory.
context: . # Context of build, this is where the project files are stored.
dockerfile: ./docker/node.dockerfile # The path to protonmail-themes Dockerfile and name of the dockerfile to be built
# Setting an image name avoids the same image being built multiple times.
image: csalmeida/protonmail-themes:latest
# Specified the name of the container, commented out to avoid name conflicts when running multiple instances of the image.
# container_name: protonmail_themes
restart: always
volumes:
# Allows changes made to project directory to be accessed by the container and persist.
- .:/var/projects/protonmail-themes
# Adds a volume to store node dependencies.
- /var/projects/protonmail-themes/node_modules
One way to know whether this is working is that a lack of a package.json file existing in the container would mean npm install wouldn’t have an entrypoint to run at and install dependencies from. Meaning that when the container runs, the context is being changed to the root of the project despite the Dockerfile being stored in a sub-directory. Essentially it seems that there’s no issue with this aspect of the configuration.
However, I did give your suggestion of moving the COPY instruction before the WORKDIR instruction a try, just to confirm if I get what you said. You are saying that the order of these statements would result in /var/projects/protonmail-themes being created before the working directory is set, thus preventing the error of the node_modules folder being missing?
Not sure if I understand completely, but I believe that the WORKDIR directive can be used with COPY and if the directory specified does not exist it will create one. I’ve made the following changes to node.dockerfile, per your suggestion (please keep in mind that the node_modules volume was also removed since it should no longer require it I assume):
# This image makes use of a Node image running on Linux Alpine
FROM node:15-alpine3.13
# Copy all project files to the container
# Files in the location of this file are copied to WORKDIR in the container
COPY . /var/projects/protonmail-themes
# A work directory is required to be used by npm install
WORKDIR /var/projects/protonmail-themes
# Makes sure npm is up to date otherwise install dependencies attempts will fail
# Install dependencies
RUN npm install -g npm && \
npm install
# The process this container should run
CMD ["npm", "start"]
The container will build successfully but the process will fail. It results in node_modules being empty which in turn will keep npm start from running.
But what if the issue is, as you mentioned, the node.dockerfile not being in the same folder as docker-compose.yml? I gave it another go by moving it to the root of the project (same directory as docker-compose.yml) and renamed it Dockerfile.
Contents of the Dockerfile:
# This image makes use of a Node image running on Linux Alpine
FROM node:15-alpine3.13
# Copy all project files to the container
# Files in the location of this file are copied to WORKDIR in the container
COPY . /var/projects/protonmail-themes
# A work directory is required to be used by npm install
WORKDIR /var/projects/protonmail-themes
# Makes sure npm is up to date otherwise install dependencies attempts will fail
# Install dependencies
RUN npm install -g npm && \
npm install
# The process this container should run
CMD ["npm", "start"]
The docker-compose.yml, again with no volume for node_modules:
version: "3"
services:
proton:
# Building a custom image described in a docker file.
build: ./
# Setting an image name avoids the same image being built multiple times.
image: csalmeida/protonmail-themes:latest
# Specified the name of the container, commented out to avoid name conflicts when running multiple instances of the image.
# container_name: protonmail_themes
restart: always
volumes:
# Allows changes made to project directory to be accessed by the container and persist.
- .:/var/projects/protonmail-themes
This will build the container successfully as well but same result, empty node_modules which is not the desired result.
I might be misunderstanding your point but if changing the order of COPY and WORKDIR was meant to create the project folder and therefore add node_modules correctly this doesn’t seem to do the trick.
My questions still remain, when npm install is ran on an image build process does it add node_modules to the image? If so, how come does it end up being empty when a container is run, unless a volume is created for it?
I understand that I might be copying unnecessary files over to the container and I will add them to .dockerignore, thanks for pointing this out!
I guess I’ve never tried otherwise, but for me WORKDIR before COPY works just fine.
While trying to solve your original problem, why not also try to speed up the process? First, only copy package.json and run npm install. This way, as long as package.json does not change, future runs will use the cached layers, speeding up things a lot! Next, copy your own source code. It seems your own code does not need a build step; if it does then you may also want to use a temporary build image in which to get the full blown node_modules and do the build, and copy only the build results into a new and much smaller target image.
Code I use for Yarn, where it seems you do not have a build step so do not need a temporary build image either:
# Temporary (partially cached) build image
FROM node:lts-alpine as build
WORKDIR /app
COPY package.json ./
COPY yarn.lock ./
RUN yarn install
# To take advantage of caching until package.json or yarn.lock changes: only now copy
# all else into the build image, and build.
# See http://bitjudo.com/blog/2014/03/13/building-efficient-dockerfiles-node-dot-js/
COPY . .
RUN yarn build
# Final target image
FROM nginx:stable-alpine
COPY --from=build /app/dist /wherever/is/your/build/result
CMD ...
Hi Arjan! Absolutely, the Dockerfile does work with no issues on that frot, however the node_modules folder would still be empty without a volume.
Thank for for the suggestions on speeding it up, will keep it in mind for when I next edit it.
Additionally, I wasn’t aware that it was possible to have multiple FROM states in a Dockerfile, this could be useful in the future.
The example files are really useful to learn more as well, appreciate it.
The WORKDIR instruction sets the working directory for any RUN, CMD, ENTRYPOINT, COPY and ADD instructions that follow it in the Dockerfile. If the WORKDIR doesn’t exist, it will be created even if it’s not used in any subsequent Dockerfile instruction.
The WORKDIR instruction can be used multiple times in a Dockerfile. If a relative path is provided, it will be relative to the path of the previous WORKDIR instruction.
In case the volume is empty and both containers have data in the target directory the first container to be run will mount its data into the volume and the other container will see that data (and not its own).
If you use either bind mounts or volumes, keep the following in mind:
If you mount an empty volume into a directory in the container in which files or directories exist, these files or directories are propagated (copied) into the volume. Similarly, if you start a container and specify a volume which does not already exist, an empty volume is created for you. This is a good way to pre-populate data that another container needs.
If you mount a bind mount or non-empty volume into a directory in the container in which some files or directories exist, these files or directories are obscured by the mount, just as if you saved files into /mnt on a Linux host and then mounted a USB drive into /mnt . The contents of /mnt would be obscured by the contents of the USB drive until the USB drive were unmounted. The obscured files are not removed or altered, but are not accessible while the bind mount or volume is mounted.
Thanks for taking the time to work this out with me Arjan! Since the thread is growing I would like to remind that the setup I have does work but I would like to know why it works the way it does.
Awesome, I hope to get to this point! I’ve built a docker image from your snippet and node_modules wasn’t a directory (logs further down). I wonder what I’m doing wrong here. For reference, in previous and current examples I’ve always removed all existing images and containers to make sure a cached version wouldn’t be used.
The snippet does make sense, the working directory is set to /app and package.json is copied to it. Dependencies install with no issues and given that /app is the current directory, a node_modules folder should be present at that time, this is the point where it fails.
Here’s the Dockerfile I used with your snippet (aside from one line, explained in the comment):
FROM node:lts-alpine as build
WORKDIR /app
COPY package.json ./
# Changed this line since I didn't have an npm.lock file
COPY package-lock.json ./
RUN npm install
RUN pwd
RUN ls -la
RUN ls -la node-modules
The WORKDIR docs make sense to me, but just to make sure I understand it, considering we have the following snippet:
WORKDIR /projects
COPY book.txt ./
This would result in book.txt being copied from the host machine directory where the Dockerfile is located to /projects in the container, is that right?
That’s cool! So that means that if I save the image the container should have the dependencies folder already, like they’re frozen in time, given I don’t build the image again. Is that right?
I don’t think this is the case since I have no other containers running, I remove them just to be sure there’s no interference when experimenting things.
This is interesting however, and it could be why node_modules is missing:
If you mount a bind mount or non-empty volume into a directory in the container in which some files or directories exist, these files or directories are obscured by the mount, just as if you saved files into /mnt on a Linux host and then mounted a USB drive into /mnt . The contents of /mnt would be obscured by the contents of the USB drive until the USB drive were unmounted. The obscured files are not removed or altered, but are not accessible while the bind mount or volume is mounted.
I think what might be happening is that node_modules does exist in the correct path but because a volume is mounted to allow changes to be made in project files between the host and the container, node_modules ends up being obscured? This is the line that does it in `docker-compose.yml:
volumes:
# Allows changes made to project directory to be accessed by the container and persist.
- .:/var/projects/protonmail-themes
Removing that line and adding a CMD to list the contents of node_modules et voilá, the folder is there and with all its contents:
If that is correct, it answers my question and helps a lot Arjan, super grateful!
Now this leads to an additional question. In the use case I presented I have n amount of project files I would like to change once the container is up and running, so I’ve setup a volume to do just that.
However, those files rely on Node dependencies, but node_modules is created at build time and then obscured when the volume that makes the project files available to the container is mounted. My question is my approach acceptable for my use case then? As in should I declare an additional volume to make node_modules is present?
- /var/projects/protonmail-themes/node_modules
Learning a lot so far, thank you, this is very insightful!
Very late reply, but I’ve stumbled upon the exact same issue where I was mounting the whole project folder into Docker image after it was built with one of the steps being npm install, overwriting node_modules in process.
Mounting additional volume for node_modules really helps making it present.
So, in my case:
