If it matters, I am using Docker version 19.03.1 build 74b1e89, Docker Desktop (community) 2.1.0.1, and macOS 10.14.6 (Mojave).
My docker file nearly completes, but right at the end, it fails with:
Step 19/21 : RUN cd /var/lib/postgresql/10/ && chown -R postgres:postgres main
---> Running in ec6785e27de5
Error processing tar file(exit status 1): write /var/lib/postgresql/10/main/base/16385/4468958.2: no space left on device
My first two questions are:
1. what is the tar file it is processing at this step?
2. what is the device it is referring to?
Perhaps with the answers to those, I could get an answer to:
3. how can I increase the amount of space on the device?
Within Docker Desktop, I have set the Disk Image Size to 208GB, 6 CPUs, 12GB of RAM, and 1GB fo Swap.
Ultimately, I want to get everything into a single Dockerfile.
What I am attempting to do is create a dockerized OpenStreetMaps tile server (https://switch2osm.org/manually-building-a-tile-server-18-04-lts/) with the North American tile data (http://download.geofabrik.de) on it.
My procedure has been to independently create the postgres database, tar’d the DB files, and then copy them into the docker image. When I deal with something small, like just the tiles for a single State, it works without issue. However, the tar’d North American data is 20GB and expanded it is ~40GB.
The point at which it fails, the DB files have already been copied over, expanded, and placed at the correct location. I am merely chown’ing them.
If I ‘docker run’ the image, I can copy over, expand, and chown without any problems. It is only when I try to do this via a Dockerfile that it fails.
I figure I a missing some nuance of how Docker works and that it is likely a simple configuration change or two that I need to make.
The details of what I am doing are:
(osm_base image Dockerfile)
FROM ubuntu:18.04
#
# Installs
#
RUN apt-get update
RUN apt-get --yes install apt-utils
RUN apt-get --yes upgrade
ENV TZ=America/New_York
ENV DEBIAN_FRONTEND=noninteractive
RUN apt-get --yes install tzdata
RUN apt-get --yes install sudo net-tools
RUN apt-get --yes install libboost-all-dev git-core tar unzip wget bzip2 build-essential autoconf libtool libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev apache2 apache2-dev libagg-dev liblua5.2-dev ttf-unifont lua5.1 liblua5.1-dev libgeotiff-epsg curl
RUN apt-get --yes install postgresql postgresql-contrib postgis postgresql-10-postgis-2.4 postgresql-10-postgis-scripts
RUN apt-get --yes install make cmake g++ libboost-dev libboost-system-dev libboost-filesystem-dev libexpat1-dev zlib1g-dev libbz2-dev libpq-dev libgeos-dev libgeos++-dev libproj-dev lua5.2 liblua5.2-dev
RUN apt-get --yes install autoconf apache2-dev libtool libxml2-dev libbz2-dev libgeos-dev libgeos++-dev libproj-dev gdal-bin libmapnik-dev mapnik-utils python-mapnik
RUN apt-get --yes install fonts-noto-cjk fonts-noto-hinted fonts-noto-unhinted ttf-unifont
RUN apt-get --yes install npm nodejs
RUN apt-get --yes install lsof
RUN npm install -g carto
RUN echo "root:12345678" | chpasswd
RUN useradd -m renderaccount
RUN echo 'renderaccount:renderpass' | chpasswd
RUN usermod -aG sudo renderaccount
#
# OSM Software
#
RUN echo "renderaccount ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
RUN echo "postgres ALL=(ALL) NOPASSWD: ALL" >> /etc/sudoers
USER postgres
RUN sudo service postgresql start &&\
createuser renderaccount &&\
createdb -E UTF8 -O renderaccount gis &&\
psql -d gis --command "CREATE EXTENSION postgis;" &&\
psql -d gis --command "CREATE EXTENSION hstore;" &&\
psql -d gis --command "ALTER TABLE geometry_columns OWNER TO renderaccount;" &&\
psql -d gis --command "ALTER TABLE spatial_ref_sys OWNER TO renderaccount;"
USER renderaccount
RUN mkdir ~/src
RUN cd ~/src &&\
git clone https://github.com/openstreetmap/osm2pgsql.git &&\
cd osm2pgsql &&\
mkdir build && cd build &&\
cmake .. &&\
make
USER root
RUN cd /home/renderaccount/src/osm2pgsql/build && make install
RUN cd /home/renderaccount/src &&\
git clone -b switch2osm https://github.com/SomeoneElseOSM/mod_tile.git &&\
cd mod_tile &&\
./autogen.sh &&\
./configure &&\
make &&\
make install &&\
make install-mod_tile &&\
ldconfig
USER renderaccount
RUN cd /home/renderaccount/src &&\
git clone https://github.com/gravitystorm/openstreetmap-carto.git &&\
cd openstreetmap-carto &&\
carto project.mml > mapnik.xml
RUN mkdir ~/data &&\
cd ~/data &&\
wget http://download.geofabrik.de/asia/azerbaijan-latest.osm.pbf
RUN cd ~/src/openstreetmap-carto/ && scripts/get-shapefiles.py
#
# OSM Sever
#
USER renderaccount
RUN sudo service postgresql start &&\
osm2pgsql -d gis --create --slim -G --hstore --tag-transform-script ~/src/openstreetmap-carto/openstreetmap-carto.lua -S ~/src/openstreetmap-carto/openstreetmap-carto.style ~/data/azerbaijan-latest.osm.pbf
USER root
COPY ./container_resources/renderd.conf /usr/local/etc/renderd.conf
RUN mkdir /var/lib/mod_tile
RUN chown renderaccount /var/lib/mod_tile
RUN mkdir /var/run/renderd
RUN chown renderaccount /var/run/renderd
COPY ./container_resources/mod_tile.conf /etc/apache2/conf-available/mod_tile.conf
RUN a2enconf mod_tile
COPY ./container_resources/000-default.conf /etc/apache2/sites-available/000-default.conf
#
# OSM Startup
#
USER renderaccount
RUN sudo cp ~/src/mod_tile/debian/renderd.init /etc/init.d/renderd
RUN sudo chmod u+x /etc/init.d/renderd
RUN sudo cp ~/src/mod_tile/debian/renderd.service /lib/systemd/system/
USER root
COPY ./container_resources/start_services.sh /start_services.sh
RUN chmod uga+x /start_services.sh
EXPOSE 80
USER renderaccount
RUN sudo locale-gen en_US.UTF-8 && sudo update-locale
This simply create the osm_base image to save me time while I try to figure out how to get the large amount of data in there.
(osm Dockerfile)
FROM osm_base:latest
USER root
COPY ./container_resources/main.tar.gz /home/renderaccount/main.tar.gz
RUN rm -rf /var/lib/postgresql/10/main
RUN cd /home/renderaccount && tar -xzvf main.tar.gz
RUN mv /home/renderaccount/main /var/lib/postgresql/10/main
RUN cd /var/lib/postgresql/10/ && chown -R postgres:postgres main
USER renderaccount
ENTRYPOINT [ "/start_services.sh" ]
It is when I try to build the osm Dockerfile that the failure occurs.
Before building the osm Dockerfile, everything has been pruned.
$ docker system df
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 2 0 5.03GB 5.03GB (100%)
Containers 0 0 0B 0B
Local Volumes 0 0 0B 0B
Build Cache 0 0 0B 0B
$ docker system info
Client:
Debug Mode: false
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 55
Server Version: 19.03.1
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 894b81a4b802e4eb2a91d1ce216b8817763c29fb
runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.184-linuxkit
Operating System: Docker Desktop
OSType: linux
Architecture: x86_64
CPUs: 6
Total Memory: 11.71GiB
Name: docker-desktop
ID: 5P2M:JRLQ:B3VE:3W62:L4XX:AR5L:PLLI:AHND:LUCQ:T2Z3:TXIJ:75LF
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 30
Goroutines: 46
System Time: 2019-08-27T19:23:57.273304321Z
EventsListeners: 2
HTTP Proxy: gateway.docker.internal:3128
HTTPS Proxy: gateway.docker.internal:3129
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine
Any thoughts on this would be appreciated.
My only guess at the moment is when building, there are intermediate containers created which I can see with ‘docker stats’ and the size of those are ~10GB which may not be big enough. However, it is unclear to me if this is a problem and, if so, how to increase their size.