Reduce size of /var/lib/overlay2 (duplicated layers)


We are deploying a python based web application on a remote server. Recently the disk usage have been increasing rapidly, and i found out that /var/lib/docker/overlay2 folder is using 80GB.

Running docker system df shows:

Images          23        23        6.272GB   2.774GB (44%)
Containers      23        23        428.1MB   0B (0%)
Local Volumes   15        13        2.238GB   76.51kB (0%)
Build Cache     79        0         6.687GB   6.687GB

Which is definitely not 80GB.

During further investigation i found that there is one folder which appears a lot of times in the /var/lib/docker/overlay2/**/diff, which is a folder within the web app container /home/app/web.

Running this command du -ch --max-depth=0 /var/lib/docker/165536.165536/overlay2/**/diff/home/app/web shows

68G total

which i think i quite high.

I am using this Dockerfile for the build:

FROM python:3.11-alpine as builder

# set work directory
WORKDIR /usr/src/app

# install psycopg2 dependencies
RUN apk update \
    && apk add postgresql-dev gcc python3-dev musl-dev

# set environment variables

# copy project
COPY poetry.lock .
COPY pyproject.toml .

# install dependencies
RUN pip install --upgrade pip
RUN pip install poetry
RUN poetry export -f requirements.txt --without dev --output requirements.txt
RUN pip install -r requirements.txt
# TODO: det her bør være i faktisk prod
# RUN python -m pip_audit -r requirements.txt

RUN pip wheel --no-cache-dir --no-deps --wheel-dir /usr/src/app/wheels -r requirements.txt
RUN poetry version -s > version.txt

FROM python:3.11-alpine

RUN mkdir -p /home/app

# create the app user
RUN addgroup -S app && adduser -S app -G app

# create the appropriate directories
ENV HOME=/home/app
ENV APP_HOME=/home/app/web

# install dependencies
RUN apk update && apk add libpq postgresql-client busybox-suid make
RUN rm -rf /var/cache/apk/*
COPY --from=builder /usr/src/app/wheels /wheels
COPY --from=builder /usr/src/app/requirements.txt .
RUN pip install --no-cache /wheels/*

# copy project

# copy version.txt, må override version.txt fra local
COPY --from=builder /usr/src/app/version.txt .

#make entrypoint executable
RUN sed -i 's/\r$//g'  $APP_HOME/
RUN chmod +x  $APP_HOME/

RUN mv ${APP_HOME}/ /
RUN sed -i 's/\r$//g'  /
RUN chmod +x  /

#create backup directory
RUN mkdir $APP_HOME/backup
RUN mkdir -p $APP_HOME/cissky/staticfiles
RUN mkdir -p $APP_HOME/cissky/mediafiles

# make documentation
# TODO: uncomment det her i faktisk produksjon
# RUN cd docs && make clean && make html && cd ..
RUN mkdir docs/html

# kjorer makemigrations i build i stedet
RUN python3 cissky/ makemigrations --noinput

# chown all the files to the app user
RUN chown -R app:app $APP_HOME

# change to the app user
USER app

# run
ENTRYPOINT ["/home/app/web/"]

Im guessing im doing something wrong with the build. Could anyone please give me some pointers?

I know its not the logs which are the issue. Ive also tried docker system prune --all --volumes, but the size will not be reduced.

Did you go into the container to check inside for the largest folders?

docker exec -it <cid> sh

Then you can run this and go deeper…

du -h --max-depth=1 | sort -hr