Strange save/load behavior

I have a very simple Dockerfile for building a custom nginx image:

FROM nginx
COPY etc/nginx/conf.d/sysadm.conf /etc/nginx/conf.d/sysadm.conf
RUN rm -f /etc/nginx/conf.d/default.conf
VOLUME /opt/cspi/pub

Here is the nginx config file /etc/nginx/nginx.conf from the base image:

user  nginx;
worker_processes  1;

error_log  /var/log/nginx/error.log warn;
pid  /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include  /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format main  '$remote_addr - $remote_user [$time_local] "$request" '
                     '$status $body_bytes_sent "$http_referer" '
                     '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;
    sendfile  on;
    keepalive_timeout  65;

    include  /etc/nginx/conf.d/*.conf;
}

I build the image with:

docker build --no-cache -t nginx_sysadm .

I run the image with:

docker run -d --name sysadm -p 80:80 nginx_sysadm

RESULT: EVERYTHING WORKS FINE.

Then I save the image with:

docker save nginx_sysadm -o nginx_sysadm.tar

And load it on a different (but identical) docker host with:

docker load -i nginx_sysadm

I run the image with:

docker run -d --name sysadm -p 80:80 nginx_sysadm

RESULT: THE CONTAINER EXITS IMMEDIATELY

When I attach STDERR, I see the following error message:

docker run -a STDERR --name sysadm -p 80:80 nginx_sysadm           
2019/09/17 21:11:20 [emerg] 1#1: open() "/etc/nginx/conf.d/default.conf" failed (2: No such file or directory) in /etc/nginx/nginx.conf:31
nginx: [emerg] open() "/etc/nginx/conf.d/default.conf" failed (2: No such file or directory) in /etc/nginx/nginx.conf:31

To troubleshoot, I copied the /etc/nginx directory from the stopped container to the local filesystem:

$ docker cp sysadm:/etc/nginx .
$ ls -l nginx/conf.d
-rw-r--r-- 1 root root 374 Sep 17 11:36 sysadm.conf
$ grep include nginx/nginx.conf
    include       /etc/nginx/mime.types;
    include /etc/nginx/conf.d/*.conf;

It is as if, within the image, the directory entry for the /etc/nginx/conf.d/default.conf remains even though the file default.conf was deleted during the build. I cannot make any sense of this.

I should add that I am very new to Docker. I’ve only been working with it for a week.

Couple more things:

– I deleted and re-creating the /etc/nginx/conf.d directory in the Dockerfile and re-built the image. Same error.

– I created ed an empty default.conf file and copied it to /etc/nginx/conf.d in the stopped container, then started the container. This worked, however, this solution is unacceptable to me.

– After getting the container to work with the empty default.conf file, I entered the container while it was running, deleted default.conf, stopped the container, then started the container. Same error as before.

– After getting the container to work with the empty default.conf file, I entered the container while it was running and attempted the following commands:

$ cd /etc/nginx
$ ls -l conf.d
-rw-r--r-- 1 root root   0 Sep 17 21:47 conf.d/default.conf
-rw-r--r-- 1 root root 374 Sep 17 21:47 conf.d/sysadm.conf
$ rm -rf conf.d
rm: cannot remove 'conf.d': Directory not empty
$ ls -l conf.d
ls: cannot access 'conf.d/default.conf': No such file or directory
ls: cannot access 'conf.d/sysadm.conf': No such file or directory
total 0
?????????? ? ? ? ?            ? default.conf
?????????? ? ? ? ?            ? sysadm.conf

I am clearly not understanding something very fundamental about Docker and the Union filesystem.