No space left on device: build works on Mac but not debian. No obvious lack of space

Hello there. I am trying to build a docker image starting with a debian image and installing on it an intel compiler. I can get the build to work no problem on my Mac, but it does not work at all on my debian machine
The error is:

dpkg: error processing archive /tmp/apt-dpkg-install-3uv2Mr/166-libalgorithm-merge-perl_0.08-3_all.deb (--unpack):
 cannot copy extracted data for './usr/share/perl5/Algorithm/Merge.pm' to '/usr/share/perl5/Algorithm/Merge.pm.dpkg-new': failed to write (No space left on device)
Selecting previously unselected package libcoarrays-dev:amd64.
Preparing to unpack .../167-libcoarrays-dev_2.9.2-3_amd64.deb ...
Unpacking libcoarrays-dev:amd64 (2.9.2-3) ...
dpkg: error processing archive /tmp/apt-dpkg-install-3uv2Mr/167-libcoarrays-dev_2.9.2-3_amd64.deb (--unpack):
 cannot copy extracted data for './usr/include/x86_64-linux-gnu/libcaf-version-def.h' to '/usr/include/x86_64-linux-gnu/libcaf-version-def.h.dpkg-new': failed to write (No space left on device)
Selecting previously unselected

It tries to do a number of other installs but they all fail with the same issue. Not sure where to even start in order to solve this one. Monitored space on machine but no issue there…
Docker version 20.10.12, build e91ed57
Debian 5.10.92-1 (2022-01-18) x86_64 GNU/Linux

Could you solve it? If not, are you sure that you checked the free space on the right device? It looks like your root partition has no free space which may not be the same partition where your other data is sotred

No. The docker directory sits on a very large nfs partition, so that’s not the problem. The /var and /tmp dirs are quite small, but monitoring them during the image build it does not show any change in usage. So either the messages are really about not being able to continue because of lack of space looking ahead, or the build uses virtual memory which in the case of this machine could be an issue.

Try to run df -h during docker build inside the build container before it fails and after it fails. Example:

RUN df -h && apt-get install ...; df -h

Note that I used “;” after the “apt-get install” so it can run df regardless of the status of the installation

@rimelek Done as you suggested, here are the results:
Before install:

Filesystem                                                                                                                       Size  Used Avail Use% Mounted on
/dev/mapper/docker-0:37-112022452-37870cd8c31c7ea909db8609019961651a32ca2cab535dbd38da6f0a5ef9c2ee                               9.8G  459M  8.8G   5% /
tmpfs                                                                                                                             64M     0   64M   0% /dev
shm                                                                                                                               64M     0   64M   0% /dev/shm
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/resolv.conf  500G  343G  158G  69% /etc/resolv.conf
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/hostname     500G  343G  158G  69% /etc/hostname
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/hosts        500G  343G  158G  69% /etc/hosts
tmpfs                                                                                                                            7.9G     0  7.9G   0% /proc/acpi
tmpfs                                                                                                                            7.9G     0  7.9G   0% /sys/firmware

After failure:

Filesystem                                                                                                                       Size  Used Avail Use% Mounted on
/dev/mapper/docker-0:37-112022452-37870cd8c31c7ea909db8609019961651a32ca2cab535dbd38da6f0a5ef9c2ee                               9.8G  6.8G  2.5G  74% /
tmpfs                                                                                                                             64M     0   64M   0% /dev
shm                                                                                                                               64M     0   64M   0% /dev/shm
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/resolv.conf  500G  342G  159G  69% /etc/resolv.conf
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/hostname     500G  342G  159G  69% /etc/hostname
nasn5:/v_botnie_data/botnie_data/docker/containers/20540138f2750bfdd82451e20f7098bd462dffd9910544cb042ccde936c219c7/hosts        500G  342G  159G  69% /etc/hosts
tmpfs                                                                                                                            7.9G     0  7.9G   0% /proc/acpi
tmpfs                                                                                                                            7.9G     0  7.9G   0% /sys/firmware

Do you use “devicemapper” as storage driver for Docker? It looks like you have 10 gigabytes maximum for your containers. If the copied file is larger than 2.5 gigabyte, it could be the problem.

Yes, I am using devicemapper. I tried to switch to overlay2 but i got error messages about some features not being supported:

Mar 04 13:48:50 srv-bengale kernel: overlayfs: upper fs does not support tmpfile.
Mar 04 13:48:50 srv-bengale kernel: overlayfs: upper fs does not support RENAME_WHITEOUT.
Mar 04 13:48:50 srv-bengale kernel: overlayfs: upper fs does not support xattr, falling back to index=off and metacopy=off.
Mar 04 13:48:50 srv-bengale kernel: overlayfs: upper fs missing required features.

Is the max size of the containers configurable?

I have never used devicemapper and it is deprecated now, but here is a tutorial that looks good

This is old, so you will need to adapt it to your Docker version.

Strange. What is your backing filesystem? The filesystem on your large partition on which your containers are created. Is it a network filesystem? NFS on NAS? Because I see “nasn5”.

The machine I am using is a Debian 5.10.92-1 (2022-01-18) virtual server (ext4 file system) with nfs mounts:
nasn5:/v_botnie_data/botnie_data on /nfs/data type nfs....
Cheers, I will try to increase the base.

Never use NFS under Docker. You can use it for individual containers (try not to use for databases or services that require fast I/O operations and locking), but the Docker data folder itself should be on a local filesystem. There might be some other network filesystems that supports everything that Docker needs, NFS is definitely not one of them and I don’t think you will find any that can be configured easily.

OK, reworked installation of Docker, now with /var/lib/docker on its own ext4 partition, and overlay2. Was able to build image. Thanx!