Docker Community Forums

Share and learn in the Docker community.

Bug in Docker? Intermediate layer not big enough

If it matters, I am using Docker version 19.03.1 build 74b1e89, Docker Desktop (community) 2.1.0.1, and macOS 10.14.6 (Mojave).

My dockerfile contains the following lines:

RUN cd /home/renderaccount && tar -xzvf main.tar.gz
RUN mv /home/renderaccount/main /var/lib/postgresql/10/main
RUN cd /var/lib/postgresql/10/ && chown -R postgres:postgres main

when executing the last line, I get the error message:

Error processing tar file(exit status 1): write /var/lib/postgresql/10/main/base/16385/4468958.2: no space left on device

If I understand the docker system correctly, during a build, an intermediate layer is constructed for each command. The docker system apparently decides, for each command, how big that layer needs to be for the command to be executed successfully. Here is a partial layer history of the failed build:

$ docker history 4fd36f61aace
IMAGE               CREATED             CREATED BY                                      SIZE                COMMENT
4fd36f61aace        12 minutes ago      /bin/sh -c mv /home/renderaccount/main /var/…   48.1GB
a5b7b9ffe84d        29 minutes ago      /bin/sh -c lsof | grep deleted || true          0B
d36f3448ddee        29 minutes ago      /bin/sh -c df -i                                0B
905e2bfa465b        29 minutes ago      /bin/sh -c df -h                                0B
648ee7f2b7a0        31 minutes ago      /bin/sh -c cd /home/renderaccount && tar -xz…   48.1GB
8161755c565d        2 hours ago         /bin/sh -c lsof | grep deleted || true          0B
2ceb6441f2bd        2 hours ago         /bin/sh -c df -i                                0B
bc31c92c4e5d        2 hours ago         /bin/sh -c df -h                                0B
a57ec209dea1        2 hours ago         /bin/sh -c rm -rf /var/lib/postgresql/10/main   0B
d855a6725873        2 hours ago         /bin/sh -c df -h                                0B
0455466d8e09        2 hours ago         /bin/sh -c #(nop) COPY file:e4b17e62219672ee…   20.8GB

So, I can see that during the expansion of the tar file and moving it, the size of the layer is 48.1GB. While the history does not show the layer during the cd & chown, there is the error message that the tar file wasn’t big enough. I assume this means that there is a tar file which is associated with the layer and that it is not or cannot be large enough. I also assume that docker is making the size decision based on the command(s) being executed.

While the cd / chown command is running, I do see an intermediate container being used while running docker stats which looks like:

CONTAINER ID a1c619f7698d
NAME goofy_dhawan
CPU % 93.43%
MEM USAGE / LIMIT 130.2MiB / 11.71GiB
MEM % 1.09%
NET I/O 1.04kB / 0B
BLOCK I/O PIDS 42.8GB / 194GB 2

I can get around this problem by doing:

RUN mv /home/renderaccount/main /var/lib/postgresql/10/main && cd /var/lib/postgresql/10/ && chown -R postgres:postgres main

In this case, it appears all of the commands are executed on a layer which is large enough and everything succeeds.

What I am interested in learning is whether or not this would be considered a docker bug or if this is intended behavior. I am not sure what other information I might be able to provided, but if more is required, please let me know.

This is related to https://forums.docker.com/t/need-to-solve-error-processing-tar-file-no-space-left-on-device/80338/2

For those interested, there is a good discussion on reddit related to this issue:

The essential points are:

  1. Everything is working as intended
  2. https://blog.mornati.net/docker-images-and-chown/ is a good article explaining some fundamentals of how Docker works
  3. Do everything possible to reduce the number of intermediate layers