Need to solve: Error processing tar file no space left on device

If it matters, I am using Docker version 19.03.1 build 74b1e89, Docker Desktop (community) 2.1.0.1, and macOS 10.14.6 (Mojave).

My docker file nearly completes, but right at the end, it fails with:

Step 19/21 : RUN cd /var/lib/postgresql/10/ && chown -R postgres:postgres main
 ---> Running in ec6785e27de5
Error processing tar file(exit status 1): write /var/lib/postgresql/10/main/base/16385/4468958.2: no space left on device

My first two questions are:

1. what is the tar file it is processing at this step?
2. what is the device it is referring to? 

Perhaps with the answers to those, I could get an answer to:

3. how can I increase the amount of space on the device?

Within Docker Desktop, I have set the Disk Image Size to 208GB, 6 CPUs, 12GB of RAM, and 1GB fo Swap.

Ultimately, I want to get everything into a single Dockerfile.

What I am attempting to do is create a dockerized OpenStreetMaps tile server (https://switch2osm.org/manually-building-a-tile-server-18-04-lts/) with the North American tile data (http://download.geofabrik.de) on it.

My procedure has been to independently create the postgres database, tar’d the DB files, and then copy them into the docker image. When I deal with something small, like just the tiles for a single State, it works without issue. However, the tar’d North American data is 20GB and expanded it is ~40GB.

The point at which it fails, the DB files have already been copied over, expanded, and placed at the correct location. I am merely chown’ing them.

If I ‘docker run’ the image, I can copy over, expand, and chown without any problems. It is only when I try to do this via a Dockerfile that it fails.

I figure I a missing some nuance of how Docker works and that it is likely a simple configuration change or two that I need to make.

The details of what I am doing are:

(osm_base image Dockerfile)

FROM ubuntu:18.04

#
# Installs
#

RUN apt-get update
RUN apt-get --yes install apt-utils
RUN apt-get --yes upgrade

ENV TZ=America/New_York
ENV DEBIAN_FRONTEND=noninteractive

RUN apt-get --yes install tzdata
RUN apt-get --yes install sudo net-tools
RUN apt-get --yes install libboost-all-dev git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg curl
RUN apt-get --yes install postgresql postgresql-contrib postgis postgresql-10-postgis-2.4 postgresql-10-postgis-scripts
RUN apt-get --yes install make cmake g++ libboost-dev libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 liblua5.2-dev
RUN apt-get --yes install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libmapnik-dev mapnik-utils python-mapnik
RUN apt-get --yes install fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont
RUN apt-get --yes install npm nodejs
RUN apt-get --yes install lsof

RUN npm install -g carto


RUN echo "root:12345678" | chpasswd
RUN useradd -m renderaccount
RUN echo 'renderaccount:renderpass' | chpasswd
RUN usermod -aG sudo renderaccount

#
# OSM Software
#

RUN echo "renderaccount ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN echo "postgres ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers

USER postgres

RUN sudo service postgresql start                                                   &&\
    createuser renderaccount                                                        &&\
    createdb -E UTF8 -O renderaccount gis                                           &&\
    psql -d gis --command "CREATE EXTENSION postgis;"                               &&\
    psql -d gis --command "CREATE EXTENSION hstore;"                                &&\
    psql -d gis --command "ALTER TABLE geometry_columns OWNER TO renderaccount;"    &&\
    psql -d gis --command "ALTER TABLE spatial_ref_sys OWNER TO renderaccount;"

USER renderaccount

RUN mkdir ~/src

RUN cd ~/src                                                    &&\
    git clone https://github.com/openstreetmap/osm2pgsql.git    &&\
    cd osm2pgsql                                                &&\
    mkdir build && cd build                                     &&\
    cmake ..                                                    &&\
    make

USER root

RUN cd /home/renderaccount/src/osm2pgsql/build && make install

RUN cd /home/renderaccount/src                                              &&\
    git clone -b switch2osm https://github.com/SomeoneElseOSM/mod_tile.git  &&\
    cd mod_tile                                                             &&\
    ./autogen.sh                                                            &&\
    ./configure                                                             &&\
    make                                                                    &&\
    make install                                                            &&\
    make install-mod_tile                                                   &&\
    ldconfig

USER renderaccount

RUN cd /home/renderaccount/src                                              &&\
    git clone https://github.com/gravitystorm/openstreetmap-carto.git       &&\
    cd openstreetmap-carto                                                  &&\
    carto project.mml > mapnik.xml

RUN mkdir ~/data                                                        &&\
    cd ~/data                                                           &&\
    wget http://download.geofabrik.de/asia/azerbaijan-latest.osm.pbf

RUN cd ~/src/openstreetmap-carto/ && scripts/get-shapefiles.py

#
# OSM Sever
#

USER renderaccount

RUN sudo service postgresql start &&\
    osm2pgsql -d gis --create --slim  -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/azerbaijan-latest.osm.pbf

USER root

COPY ./container_resources/renderd.conf /usr/local/etc/renderd.conf

RUN mkdir /var/lib/mod_tile
RUN chown renderaccount /var/lib/mod_tile
RUN mkdir /var/run/renderd
RUN chown renderaccount /var/run/renderd

COPY ./container_resources/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf

RUN a2enconf mod_tile

COPY ./container_resources/000-default.conf /etc/apache2/sites-available/000-default.conf

#
# OSM Startup
#

USER renderaccount

RUN sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
RUN sudo chmod u+x /etc/init.d/renderd
RUN sudo cp ~/src/mod_tile/debian/renderd.service /lib/systemd/system/

USER root

COPY ./container_resources/start_services.sh /start_services.sh
RUN chmod uga+x /start_services.sh

EXPOSE 80

USER renderaccount

RUN sudo locale-gen en_US.UTF-8 && sudo update-locale

This simply create the osm_base image to save me time while I try to figure out how to get the large amount of data in there.

(osm Dockerfile)

FROM osm_base:latest
USER root
COPY ./container_resources/main.tar.gz /home/renderaccount/main.tar.gz
RUN rm -rf /var/lib/postgresql/10/main
RUN cd /home/renderaccount && tar -xzvf main.tar.gz
RUN mv /home/renderaccount/main /var/lib/postgresql/10/main
RUN cd /var/lib/postgresql/10/ && chown -R postgres:postgres main
USER renderaccount
ENTRYPOINT [ "/start_services.sh" ]

It is when I try to build the osm Dockerfile that the failure occurs.

Before building the osm Dockerfile, everything has been pruned.

$ docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              2                   0                   5.03GB              5.03GB (100%)
Containers          0                   0                   0B                  0B
Local Volumes       0                   0                   0B                  0B
Build Cache         0                   0                   0B                  0B

$ docker system info
Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 55
 Server Version: 19.03.1
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
 runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 4.9.184-linuxkit
 Operating System: Docker Desktop
 OSType: linux
 Architecture: x86_64
 CPUs: 6
 Total Memory: 11.71GiB
 Name: docker-desktop
 ID: 5P2M:JRLQ:B3VE:3W62:L4XX:AR5L:PLLI:AHND:LUCQ:T2Z3:TXIJ:75LF
 Docker Root Dir: /var/lib/docker
 Debug Mode: true
  File Descriptors: 30
  Goroutines: 46
  System Time: 2019-08-27T19:23:57.273304321Z
  EventsListeners: 2
 HTTP Proxy: gateway.docker.internal:3128
 HTTPS Proxy: gateway.docker.internal:3129
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
 Product License: Community Engine

Any thoughts on this would be appreciated.

My only guess at the moment is when building, there are intermediate containers created which I can see with ‘docker stats’ and the size of those are ~10GB which may not be big enough. However, it is unclear to me if this is a problem and, if so, how to increase their size.

I did take a look at the docker.log during the failure and this is the last few lines of the output:

2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.046186305Z" level=debug msg="Revoking external connectivity on endpoint crazy_bardeen (d584a0172f219a57285b4ae9afba83e7453a8ea65ccb2ccf42121065e81253ff)"docker time="2019-08-28T20:30:23.056603310Z" level=info msg="shim containerd-shim started" address="/containerd-shim/moby/3c3d3f2eaf8dae02416ce94043f358a44ed9d268e252019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.047589045Z" level=debug msg="DeleteConntrackEntries purged ipv4:0, ipv6:0"
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.255553168Z" level=debug msg="Releasing addresses for endpoint crazy_bardeen's interface on network bridge"
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.255626188Z" level=debug msg="ReleaseAddress(LocalDefault/172.17.0.0/16, 172.17.0.2)"
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.255659874Z" level=debug msg="Released address PoolID:LocalDefault/172.17.0.0/16, Address:172.17.0.2 Sequence:App: ipam/default/data, ID: LocalDefault/172.17.0.0/16, DBIndex: 0x0, Bits: 65536, Unselected: 65532, Sequence: (0xe0000000, 1)->(0x0, 2046)->(0x1, 1)->end Curr:3"
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.295226278Z" level=warning msg="3c3d3f2eaf8dae02416ce94043f358a44ed9d268e2588adf40b1373660a7b188 cleanup: failed to unmount IPC: umount /var/lib/docker/containers/3c3d3f2eaf8dae02416ce94043f358a44ed9d268e2588adf40b1373660a7b188/mounts/shm, flags: 0x2: no such file or directory"
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.346734517Z" level=debug msg="event published" ns=moby topic="/containers/delete" type=containerd.events.ContainerDelete
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.353600197Z" level=debug msg="Tar with options on /var/lib/docker/overlay2/fa6512775cb833a1d84a57464648bca11fb0a7d35d3dc5bc0fac1823118297e1/diff" storage-driver=overlay278fed52) entered forwarding state
2019-08-28T20:32:56Z docker time="2019-08-28T20:32:56.362453978Z" level=debug msg="Applying tar in /var/lib/docker/overlay2/ac815fc2ee2f5d0d3dda31556450ef99c90c092aacc7e08ddc4ad1a1207893d3/diff" storage-driver=overlay2
2019-08-28T20:30:23Z docker time="2019-08-28T20:30:23.861016004Z" level=debug msg="event published" ns=moby topic="/tasks/create" type=containerd.events.TaskCreate
2019-08-28T20:30:23Z docker time="2019-08-28T20:30:23.861710045Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/create
2019-08-28T20:30:23Z docker time="2019-08-28T20:30:23.883923319Z" level=debug msg="event published" ns=moby topic="/tasks/start" type=containerd.events.TaskStart
2019-08-28T20:30:23Z docker time="2019-08-28T20:30:23.884576357Z" level=debug msg=event module=libcontainerd namespace=moby topic=/tasks/start
2019-08-28T20:30:23Z docker time="2019-08-28T20:30:23.904724465Z" level=debug msg="Calling GET /v1.40/containers/3c3d3f2eaf8dae02416ce94043f358a44ed9d268e2588adf40b1373660a7b188/json"
2019-08-28T20:37:55Z docker time="2019-08-28T20:37:55.418590600Z" level=debug msg="Cleaning up layer ac815fc2ee2f5d0d3dda31556450ef99c90c092aacc7e08ddc4ad1a1207893d3: Error processing tar file(exit status 1): write /var/lib/postgresql/10/main/base/16385/4468958.2: no space left on device"

I assume the tar file being referred to in the message “Error processing tar file” is the one in the message which says:

Applying tar in /var/lib/docker/overlay2/ac815fc2ee2f5d0d3dda31556450ef99c90c092aacc7e08ddc4ad1a1207893d3/diff" storage-driver=overlay2

So, this tar file is apparently to small for the data I am processing. How can I increase the size of it?