Getting a strange result when building a docker host using open-embedded/yocto

spierepf · February 28, 2018, 7:52pm

I am using OpenEmbedded/yocto to build an image for an arm-based docker host. I have it mostly working, but I think there is a problem with how the resulting host handles filesystem overlays.

I’ve written a simple shell script that highlights my problem:

docker build -t test - <<"EOF"
FROM alpine
RUN touch /usr/bin/bogus
EOF

docker run -it --rm test sh -c "whoami && echo PASS || echo FAIL"

docker rmi test

Running this test on my docker-enabled ubuntu desktop causes the “PASS” message to appear, while running this test on my yocto-based host causes “FAIL”.

The test tries to build a new image based on alpine (which is available for both x86_64 and armhf architectures) and adds a file to /usr/bin. For some reason, on my yocto-based host, this causes containers spun from the image to only include the new file in /usr/bin.

spierepf · February 28, 2018, 8:07pm

Here is my docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 2
Server Version: 1.13.0
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 03e5862ec0d8d3b3f750e19fca3ee367e13c090e
runc version: 2f7393a47307a16f8cee44a37b262e8b81021e3e
init version: N/A (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Kernel Version: 4.0.0-ADI-1.2.0-yocto-standard
Operating System: <unknown>
OSType: linux
Architecture: armv7l
CPUs: 1
Total Memory: 216 MiB
Name: adzs-sc589-ezlite
ID: LRTT:UTER:6YTZ:NZ5A:OFAF:R56E:JYKK:YGOC:VUJG:YHNU:RDI4:WLAP
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

could it have something to do with:

Native Overlay Diff: false

And if so, any pointers to a kernel or some other config that I might need to add or change?

sdetweil · February 28, 2018, 8:13pm

what does docker logs show before you delete the container and image? (take off --rm)

spierepf · February 28, 2018, 8:35pm

This is what dockerd outputs while I run the test:

time="2018-02-28T20:26:48.194654352Z" level=warning msg="Your kernel does not support cgroup memory limit" 
time="2018-02-28T20:26:48.195381988Z" level=warning msg="Your kernel does not support cgroup cfs period" 
time="2018-02-28T20:26:48.195687677Z" level=warning msg="Your kernel does not support cgroup cfs quotas" 
time="2018-02-28T20:26:48.196235739Z" level=warning msg="Unable to find blkio cgroup in mounts" 
time="2018-02-28T20:26:48.197232681Z" level=warning msg="mountpoint for pids not found" 
time="2018-02-28T20:26:49.135865419Z" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found" 
time="2018-02-28T20:26:49.587234779Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 2b2dafaab209c84af8975d61a18b7112fa6dc72a812d0d1895a0936dde81ec39 does not exist\\no
ne or more of the container deletions failed\\n\"" 
time="2018-02-28T20:26:52.479220290Z" level=warning msg="Your kernel does not support cgroup memory limit" 
time="2018-02-28T20:26:52.480047294Z" level=warning msg="Your kernel does not support cgroup cfs period" 
time="2018-02-28T20:26:52.480388121Z" level=warning msg="Your kernel does not support cgroup cfs quotas" 
time="2018-02-28T20:26:52.480872210Z" level=warning msg="Unable to find blkio cgroup in mounts" 
time="2018-02-28T20:26:52.481915819Z" level=warning msg="mountpoint for pids not found" 
time="2018-02-28T20:26:53.370697579Z" level=error msg="containerd: notify OOM events" error="cgroup path for memory not found" 
time="2018-02-28T20:26:53.832641081Z" level=error msg="containerd: deleting container" error="exit status 1: \"container 8595d86098f1f3754bd13505257e8f0aac7d17b20c3e9016149080395c79106e does not exist\\no
ne or more of the container deletions failed\\n\""

sdetweil · February 28, 2018, 8:44pm

not what i wanted

docker logs container_id

spierepf · March 1, 2018, 12:56pm

First off, let me thank you for taking the time to help me look into this. I’ve changed my script slightly to get the docker logs (sorry, I didn’t know that was a thing):

# cat test.sh 
docker build -t test - <<"EOF"
FROM alpine
RUN touch /usr/bin/bogus
EOF

docker run -it --name=test_container test sh -c "whoami && echo PASS || echo FAIL"

docker logs test_container

docker rm test_container

docker rmi test

And here is the corresponding output on my yocto host:

# ./test.sh 
Sending build context to Docker daemon 2.048 kB
Step 1/2 : FROM alpine
 ---> fdf451efac74
Step 2/2 : RUN touch /usr/bin/bogus
 ---> Running in b3e9dc99e2a4
[64241.650177] device veth159b0de entered promiscuous mode
[64243.818746] eth0: renamed from veth3515a72
[64243.829781] docker0: port 1(veth159b0de) entered forwarding state
[64243.841736] docker0: port 1(veth159b0de) entered forwarding state
[64244.446189] veth3515a72: renamed from eth0
[64244.554645] docker0: port 1(veth159b0de) entered disabled state
[64244.567794] device veth159b0de left promiscuous mode
[64244.577448] docker0: port 1(veth159b0de) entered disabled state
 ---> 4730c8befb08
Removing intermediate container b3e9dc99e2a4
Successfully built 4730c8befb08
[64246.916379] device veth53c589b entered promiscuous mode
[64248.391182] eth0: renamed from veth7ec1c02
[64248.402141] docker0: port 1(veth53c589b) entered forwarding state
[64248.414092] docker0: port 1(veth53c589b) entered forwarding state
sh: whoami: not found
FAIL
[64249.042725] veth7ec1c02: renamed from eth0
[64249.158301] docker0: port 1(veth53c589b) entered disabled state
[64249.180823] device veth53c589b left promiscuous mode
[64249.190532] docker0: port 1(veth53c589b) entered disabled state
sh: whoami: not found
FAIL
test_container
Untagged: test:latest
Deleted: sha256:4730c8befb081bacab45150fe5e5a701305752c0bd946138a10fc30471b82f99
Deleted: sha256:f66d21a4d8f8c97b45ce01e93d04eb54a6b5bd5b4111817d8a1e87afb0f45ba0

sdetweil · March 1, 2018, 12:58pm

there is the error, whoami is not installed or accessable

spierepf · March 1, 2018, 1:25pm

Sorry, I think you misunderstand. The /usr/bin/whoami command is part of the alpine image. However, if I use docker build to create a new image based on alpine and create a new file in /usr/bin the new image will only have the new file in /usr/bin.

To be clear, if I change my script and comment out the RUN touch /usr/bin/bogus from the docker build here document:

# cat test.sh 
docker build -t test - <<"EOF"
FROM alpine
#RUN touch /usr/bin/bogus
EOF

docker run -it --name=test_container test sh -c "whoami && echo PASS || echo FAIL"

docker logs test_container

docker rm test_container

docker rmi test

then running the test gets the PASS result:

# ./test.sh 
Sending build context to Docker daemon 2.048 kB
Step 1/1 : FROM alpine
 ---> fdf451efac74
Successfully built fdf451efac74
[66011.435939] device veth6c9847b entered promiscuous mode
[66013.224262] eth0: renamed from veth15f79b1
[66013.234237] docker0: port 1(veth6c9847b) entered forwarding state
[66013.246175] docker0: port 1(veth6c9847b) entered forwarding state
root
PASS
[66014.299353] docker0: port 1(veth6c9847b) entered disabled state
[66014.311672] veth15f79b1: renamed from eth0
[66014.419984] docker0: port 1(veth6c9847b) entered disabled state
[66014.433653] device veth6c9847b left promiscuous mode
[66014.443346] docker0: port 1(veth6c9847b) entered disabled state
root
PASS
test_container
Untagged: test:latest

sdetweil · March 1, 2018, 1:43pm

ok, before and after touch, run ls -laF /usr and ls -laF ./usr/bin

is this a build time problem or run time problem?

if you built the image on a different platform and then executed it here, would it have the same problem?

spierepf · March 1, 2018, 1:56pm

Here is the updated output:

# ./test.sh                                                                                                                                                                 
Sending build context to Docker daemon 2.048 kB
Step 1/6 : FROM alpine
 ---> fdf451efac74
Step 2/6 : RUN ls -laF /usr
 ---> Running in 6a4d9af1fce4
[67981.469455] device vethce62daa entered promiscuous mode
[67983.316391] eth0: renamed from veth74d9609
[67983.327404] docker0: port 1(vethce62daa) entered forwarding state
[67983.339355] docker0: port 1(vethce62daa) entered forwarding state
total 28
drwxr-xr-x    7 root     root          4096 Nov 30 21:52 ./
drwxr-xr-x    1 root     root          4096 Mar  1 13:46 ../
drwxr-xr-x    2 root     root          4096 Nov 30 21:52 bin/
drwxr-xr-x    2 root     root          4096 Nov 30 21:52 lib/
drwxr-xr-x    5 root     root          4096 Nov 30 21:52 local/
drwxr-xr-x    2 root     root          4096 Nov 30 21:52 sbin/
drwxr-xr-x    5 root     root          4096 Nov 30 21:52 share/
[67983.955394] veth74d9609: renamed from eth0
[67984.056171] docker0: port 1(vethce62daa) entered disabled state
[67984.069552] device vethce62daa left promiscuous mode
[67984.079247] docker0: port 1(vethce62daa) entered disabled state
 ---> cc1c8e4b61b4
Removing intermediate container 6a4d9af1fce4
Step 3/6 : RUN ls -laF ./usr/bin
 ---> Running in 491ecdcb787b
[67986.013628] device vethe23d66b entered promiscuous mode
[67987.498842] eth0: renamed from vetha5143da
[67987.510491] docker0: port 1(vethe23d66b) entered forwarding state
[67987.522444] docker0: port 1(vethe23d66b) entered forwarding state
total 168
drwxr-xr-x    2 root     root          4096 Nov 30 21:52 ./
drwxr-xr-x    7 root     root          4096 Nov 30 21:52 ../
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 [ -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 [[ -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 awk -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 basename -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 beep -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 blkdiscard -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 bunzip2 -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 bzcat -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 bzip2 -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cal -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 chvt -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cksum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 clear -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cmp -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 comm -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cpio -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 crontab -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cryptpw -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 cut -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 dc -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 deallocvt -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 diff -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 dirname -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 dos2unix -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 du -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 dumpleases -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 eject -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 env -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 expand -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 expr -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 factor -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 fallocate -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 find -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 flock -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 fold -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 N[67988.175376] vetha5143da: renamed from eth0
ov 30 21:52 free -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 fuser -> /bin/busybox*
-rwxr-xr-x    1 root     root         25316 Nov 20 08:28 getconf*
-rwxr-xr-x    1 root     root         36860 Nov 20 08:28 getent*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 groups -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 hd -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 head -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 hexdump -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 hostid -> /bin/busybox*
-rwxr-xr-x    1 root     root         19088 Nov 20 08:28 iconv*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 id -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 install -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 ipcrm -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 ipcs -> /bin/busybox*
lrwxrwxrwx    1 root     root        [67988.488806] docker0: port 1(vethe23d66b) entered disabled state
    12 Nov 30 21:52 killall -> /bin/busybox*[67988.583551] device vethe23d66b left promiscuous mode

lrwxrwxrwx    1 root     root            28 Nov 30 21:52 ldd -> ../../lib/ld-musl-armhf.so.1*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 less -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 logger -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 lsof -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 lsusb -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 lzcat -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 lzma -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 lzopcat -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 md5sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 mesg -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 microcom -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 mkfifo -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 mkpasswd -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nc -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nl -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nmeter -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nohup -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nproc -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nsenter -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 nslookup -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 od -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 openvt -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 passwd -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 paste -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 patch -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pgrep -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pkill -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pmap -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 printf -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pscan -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pstree -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 pwdx -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 readlink -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 realpath -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 renice -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 reset -> /bin/busybox*
lrwxrwxrwx    1 root     root[67988.599951] docker0: port 1(vethe23d66b) entered disabled state
            12 Nov 30 21:52 resize -> /bin/busybox*
-rwxr-xr-x    1 root     root         75036 Oct 26 10:33 scanelf*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 seq -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 setkeycodes -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 setsid -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sha1sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sha256sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sha3sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sha512sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 showkey -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 shred -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 shuf -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 smemcap -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sort -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 split -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 ssl_client -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 strings -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 sum -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 tac -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 tail -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 tee -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 test -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 time -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 timeout -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 top -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 tr -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 traceroute -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 traceroute6 -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 truncate -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 tty -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 ttysize -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 udhcpc6 -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unexpand -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 uniq -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unix2dos -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unlink -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unlzma -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unlzop -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unshare -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unxz -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 unzip -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 uptime -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 uudecode -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 uuencode -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 vi -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 vlock -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 volname -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 wc -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 wget -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 which -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 whoami -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 whois -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 xargs -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 xxd -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 xzcat -> /bin/busybox*
lrwxrwxrwx    1 root     root            12 Nov 30 21:52 yes -> /bin/busybox*
 ---> 45a3533f17c3
Removing intermediate container 491ecdcb787b
Step 4/6 : RUN touch /usr/bin/bogus
 ---> Running in 27991aca58a6
[67991.055294] device veth1f721f5 entered promiscuous mode
[67992.216304] eth0: renamed from vethfdba6db
[67992.226928] docker0: port 1(veth1f721f5) entered forwarding state
[67992.238882] docker0: port 1(veth1f721f5) entered forwarding state
[67992.837461] vethfdba6db: renamed from eth0
[67992.936056] docker0: port 1(veth1f721f5) entered disabled state
[67992.950007] device veth1f721f5 left promiscuous mode
[67992.959710] docker0: port 1(veth1f721f5) entered disabled state
 ---> 3ebe40d49f23
Removing intermediate container 27991aca58a6
Step 5/6 : RUN ls -laF /usr
 ---> Running in 4bfbd5620062
[67994.814704] device veth8697e21 entered promiscuous mode
[67995.966651] eth0: renamed from vethccf1fef
[67995.977507] docker0: port 1(veth8697e21) entered forwarding state
[67995.989452] docker0: port 1(veth8697e21) entered forwarding state
total 12
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ./
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ../
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 bin/
[67996.588024] vethccf1fef: renamed from eth0
[67996.695268] docker0: port 1(veth8697e21) entered disabled state
[67996.708704] device veth8697e21 left promiscuous mode
[67996.718406] docker0: port 1(veth8697e21) entered disabled state
 ---> 4efa7f9034d8
Removing intermediate container 4bfbd5620062
Step 6/6 : RUN ls -laF ./usr/bin
 ---> Running in 5bfd74c55016
[67998.559566] device veth64fc27e entered promiscuous mode
[67999.745784] eth0: renamed from veth11f421b
[67999.757816] docker0: port 1(veth64fc27e) entered forwarding state
[67999.769777] docker0: port 1(veth64fc27e) entered forwarding state
total 8
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ./
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ../
-rw-r--r--    1 root     root             0 Mar  1 13:47 bogus
[68000.374655] veth11f421b: renamed from eth0
[68000.473353] docker0: port 1(veth64fc27e) entered disabled state
[68000.486359] device veth64fc27e left promiscuous mode
[68000.496047] docker0: port 1(veth64fc27e) entered disabled state
 ---> 17710b8b2feb
Removing intermediate container 5bfd74c55016
Successfully built 17710b8b2feb
[68002.432025] device vethfa463f1 entered promiscuous mode
[68003.554257] eth0: renamed from vethb6d46b6
[68003.566017] docker0: port 1(vethfa463f1) entered forwarding state
[68003.577975] docker0: port 1(vethfa463f1) entered forwarding state
total 12
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ./
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ../
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 bin/
total 8
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ./
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ../
-rw-r--r--    1 root     root             0 Mar  1 13:47 bogus
sh: whoami: not found
FAIL
[68004.221020] vethb6d46b6: renamed from eth0
[68004.347390] docker0: port 1(vethfa463f1) entered disabled state
[68004.372685] device vethfa463f1 left promiscuous mode
[68004.382386] docker0: port 1(vethfa463f1) entered disabled state
total 12
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ./
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ../
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 bin/
total 8
drwxr-xr-x    1 root     root          4096 Mar  1 13:47 ./
drwxr-xr-x    1 root     root          4096 Nov 30 21:52 ../
-rw-r--r--    1 root     root             0 Mar  1 13:47 bogus
sh: whoami: not found
FAIL
test_container
Untagged: test:latest
Deleted: sha256:17710b8b2feb28e28bf80aa3a84ed899cdfe34c9ca2e2568a19ec2ce399711f7
Deleted: sha256:4efa7f9034d8fe536aaf24f1b39547d69ebaab1eb106e98e951d1aff3a0e7dda
Deleted: sha256:3ebe40d49f230f56762073c540b90e67f3daeff761ff3e9c9cbe2a601d47967a
Deleted: sha256:a0e23880616643ae0f8a5ddab721690b81c12a818395bfb432f96092c6b4361b
Deleted: sha256:45a3533f17c346ba2797a6fb93614b95cc117b02e007251345fc43a987dd896d
Deleted: sha256:cc1c8e4b61b4b0da45474760a6ef61f4f9b41a0359a9718de0edde7608648b15

Here is my updated test script:

# cat test.sh 
docker build -t test - <<"EOF"
FROM alpine
RUN ls -laF /usr
RUN ls -laF ./usr/bin
RUN touch /usr/bin/bogus
RUN ls -laF /usr
RUN ls -laF ./usr/bin
EOF

docker run -it --name=test_container test sh -c "ls -laF /usr ; ls -laF ./usr/bin ; whoami && echo PASS || echo FAIL"

docker logs test_container

docker rm test_container

docker rmi test

sdetweil · March 1, 2018, 2:01pm

yep, looks like /usr/bin was a mounted volume from a busybox, and the touch forced it to be local, dropping the link

spierepf · March 1, 2018, 2:37pm

I may be misunderstanding but, debian doesn’t use busybox so I tried replacing the alpine source image with debian. Same result:

# ./test.sh 
Sending build context to Docker daemon 2.048 kB
Step 1/6 : FROM debian
latest: Pulling from library/debian
0fe3809766af: Pull complete 
Digest: sha256:4fcd8c0b6f5e3bd44a3e63be259fd0c038476d432953d449ef34aedf16def331
Status: Downloaded newer image for debian:latest
 ---> 8f4445fe8219
Step 2/6 : RUN ls -laF /usr
 ---> Running in c7fec2d678ed
[ 1070.531935] device veth98a8356 entered promiscuous mode
[ 1072.063422] eth0: renamed from veth5d75806
[ 1072.074115] docker0: port 1(veth98a8356) entered forwarding state
[ 1072.086077] docker0: port 1(veth98a8356) entered forwarding state
total 40
drwxr-xr-x 10 root root  4096 Feb 13 00:00 ./
drwxr-xr-x  1 root root  4096 Mar  1 14:26 ../
drwxr-xr-x  2 root root  4096 Feb 13 00:00 bin/
drwxr-xr-x  2 root root  4096 Nov 19 15:25 games/
drwxr-xr-x  2 root root  4096 Nov 19 15:25 include/
drwxr-xr-x 13 root root  4096 Feb 13 00:00 lib/
drwxrwsr-x 10 root staff 4096 Feb 13 00:00 local/
drwxr-xr-x  2 root root  4096 Feb 13 00:00 sbin/
drwxr-xr-x 31 root root  4096 Feb 13 00:00 share/
drwxr-xr-x  2 root root  4096 Nov 19 15:25 src/
[ 1072.868481] veth5d75806: renamed from eth0
[ 1072.969729] docker0: port 1(veth98a8356) entered disabled state
[ 1072.982659] device veth98a8356 left promiscuous mode
[ 1072.992348] docker0: port 1(veth98a8356) entered disabled state
 ---> a8d0ec2e1053
Removing intermediate container c7fec2d678ed
Step 3/6 : RUN ls -laF ./usr/bin
 ---> Running in fa9af5f39152
[ 1079.753453] device veth656f43a entered promiscuous mode
[ 1080.927835] eth0: renamed from veth784e7d4
[ 1080.938619] docker0: port 1(veth656f43a) entered forwarding state
[ 1080.950573] docker0: port 1(veth656f43a) entered forwarding state
total 11508
drwxr-xr-x  2 root root      4096 Feb 13 00:00 ./
drwxr-xr-x 10 root root      4096 Feb 13 00:00 ../
-rwxr-xr-x  1 root root     47040 Feb 22  2017 [*
-rwxr-xr-x  1 root root     18152 Mar 22  2017 addpart*
-rwxr-xr-x  1 root root     13820 Sep 13 16:47 apt*
-rwxr-xr-x  1 root root     71248 Sep 13 16:47 apt-cache*
-rwxr-xr-x  1 root root     17968 Sep 13 16:47 apt-cdrom*
-rwxr-xr-x  1 root root     17916 Sep 13 16:47 apt-config*
-rwxr-xr-x  1 root root     34352 Sep 13 16:47 apt-get*
-rwxr-xr-x  1 root root     26269 Sep 13 16:47 apt-key*
-rwxr-xr-x  1 root root     34352 Sep 13 16:47 apt-mark*
-rwxr-xr-x  1 root root     26548 Feb 22  2017 arch*
lrwxrwxrwx  1 root root        21 Feb 13 00:00 awk -> /etc/alternatives/awk*
-rwxr-xr-x  1 root root     79868 Feb 22  2017 b2sum*
-rwxr-xr-x  1 root root     34772 Feb 22  2017 base32*
-rwxr-xr-x  1 root root     34772 Feb 22  2017 base64*
-rwxr-xr-x  1 root root     26544 Feb 22  2017 basename*
-rwxr-xr-x  1 root root      7139 May 15  2017 bashbug*
lrwxrwxrwx  1 root root         3 Sep  7 17:05 captoinfo -> tic*
-rwxr-xr-x  1 root root      3303 Jun 15  2017 catchsegv*
-rwxr-sr-x  1 root shadow   57832 May 17  2017 chage*
-rwxr-xr-x  1 root root      9676 Feb  1  2017 chattr*
-rwxr-xr-x  1 root root     59456 Feb 22  2017 chcon*
-rwsr-xr-x  1 root root     44364 May 17  2017 chfn*
-rwxr-xr-x  1 root root     30568 Mar 22  2017 chrt*
-rwsr-xr-x  1 root root     35492 May 17  2017 chsh*
-rwxr-xr-x  1 root root     30652 Feb 22  2017 cksum*
-rwxr-xr-x  1 root root      5576 Sep  7 17:05 clear*
-rwxr-xr-x  1 root root      9856 May 15  2017 clear_console*
-rwxr-xr-x  1 root root     38780 Jan  9  2017 cmp*
-rwxr-xr-x  1 root root     34760 Feb 22  2017 comm*
-rwxr-xr-x  1 root root     43020 Feb 22  2017 csplit*
lrwxrwxrwx  1 root root         6 Nov 24 09:22 ctstat -> lnstat*
-rwxr-xr-x  1 root root     34800 Feb 22  2017 cut*
-rwxr-xr-x  1 root root     20142 May  2  2017 deb-systemd-helper*
-rwxr-xr-x  1 root root      4507 May  2  2017 deb-systemd-invoke*
-rwxr-xr-x  1 root root      2859 May 21  2017 debconf*
-rwxr-xr-x  1 root root     11541 May 21  2017 debconf-apt-progress*
-rwxr-xr-x  1 root root       608 May 21  2017 debconf-communicate*
-rwxr-xr-x  1 root root      1719 May 21  2017 debconf-copydb*
-rwxr-xr-x  1 root root       647 May 21  2017 debconf-escape*
-rwxr-xr-x  1 root root      2935 May 21  2017 debconf-set-selections*
-rwxr-xr-x  1 root root      1827 May 21  2017 debconf-show*
-rwxr-xr-x  1 root root     18152 Mar 22  2017 delpart*
-rwxr-xr-x  1 root root    129388 Jan  9  2017 diff*
-rwxr-xr-x  1 root root     51156 Jan  9  2017 diff3*
-rwxr-xr-x  1 root root     38864 Feb 22  2017 dircolors*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 dirname*
-rwxr-xr-x  1 root root    259832 May 17  2017 dpkg*
-rwxr-xr-x  1 root root    124584 May 17  2017 dpkg-deb*
-rwxr-xr-x  1 root root    116416 May 17  2017 dpkg-divert*
-rwxr-xr-x  1 root root     19030 May 17  2017 dpkg-maintscript-helper*
-rwxr-xr-x  1 root root    120508 May 17  2017 dpkg-query*
-rwxr-xr-x  1 root root     99992 May 17  2017 dpkg-split*
-rwxr-xr-x  1 root root     58972 May 17  2017 dpkg-statoverride*
-rwxr-xr-x  1 root root     54872 May 17  2017 dpkg-trigger*
-rwxr-xr-x  1 root root    100516 Feb 22  2017 du*
-rwxr-xr-x  1 root root     26556 Feb 22  2017 env*
-rwxr-xr-x  1 root root     30680 Feb 22  2017 expand*
-rwxr-sr-x  1 root shadow   18000 May 17  2017 expiry*
-rwxr-xr-x  1 root root     38860 Feb 22  2017 expr*
-rwxr-xr-x  1 root root     79856 Feb 22  2017 factor*
-rwxr-xr-x  1 root root     18008 May 17  2017 faillog*
-rwxr-xr-x  1 root root     26408 Mar 22  2017 fallocate*
-rwxr-xr-x  1 root root    191260 Feb 18  2017 find*
-rwxr-xr-x  1 root root     26508 Mar 22  2017 flock*
-rwxr-xr-x  1 root root     38860 Feb 22  2017 fmt*
-rwxr-xr-x  1 root root     30668 Feb 22  2017 fold*
-rwxr-xr-x  1 root root     18056 Jun 15  2017 getconf*
-rwxr-xr-x  1 root root     18528 Jun 15  2017 getent*
-rw[ 1081.643443] veth784e7d4: renamed from eth0
xr-xr-x  1 root root     14056 Mar 22  2017 getopt*
-rwsr-xr-x  1 root root     65956 May 17  2017 gpasswd*
-rwxr-xr-x  1 root root    378656 Sep 18 20:41 gpgv*
-rwxr-xr-x  1 root root     30664 Feb 22  2017 groups*
-rwxr-xr-x  1 root root     38864 Feb 22  2017 head*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 hostid*
-rwxr-xr-x  1 root root     47032 Jun 15  2017 iconv*
-rwxr-xr-x  1 root root     34804 Feb 22  2017 id*
-rwxr-xr-x  1 root root     54776 Sep  7 17:05 infocmp*
lrwxrwxrwx  1 root root         3 Sep  7 17:05 infotocap -> tic*
-rwxr-xr-x  1 root root    121232 Feb 22  2017 install*
-rwxr-xr-x  1 root root     22308 Mar 22  2017 ionice*
-rwxr-xr-x  1 root root     22396 Mar 22  2017 ipcmk*
-rwxr-xr-x  1 root root     26396 Mar 22  2017 ipcrm*
-rwxr-xr-x  1 root root     46956 Mar 22  2017 ipcs*
-rwxr-xr-x  1 root root      9768 Apr  2  2017 ischroot*
-rwxr-xr-x  1 root root     38896 Feb 22  2017 join*
-rwxr-xr-x  1 root root     38804 Mar 22  2017 last*
lrwxrwxrwx  1 root root         4 Mar 22  2017 lastb -> last*
-rwxr-xr-x  1 root root     17916 May 17  2017 lastlog*
-rwxr-xr-x  1 root root      5315 Jun 15  2017 ldd*
-rwxr-xr-x  1 root root      9888 Mar 22  2017 line*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 link*
lrwxrwxrwx  1 root root         7 Mar 22  2017 linux32 -> setarch*
lrwxrwxrwx  1 root root         7 Mar 22  2017 linux64 -> setarch*
-rwxr-xr-x  1 root root     18268 Nov 24 09:22 lnstat*
-rwxr-xr-x  1 root root     31908 Jun 15  2017 locale*
-rwxr-xr-x  1 root root    281084 Jun 15  2017 localedef*
-rwxr-xr-x  1 root root     39084 Mar 22  2017 logger*
-rwxr-xr-x  1 root root     26544 Feb 22  2017 logname*
-rwxr-xr-x  1 root root      9676 Feb  1  2017 lsattr*
-rwxr-xr-x  1 root root     55336 Mar 22  2017 lscpu*
-rwxr-xr-x  1 root root     59412 Mar 22  2017 lsipc*
-rwxr-xr-x  1 root root     30844 Mar 22  2017 lslocks*
-rwxr-xr-x  1 root root     51324 Mar 22  2017 lslogins*
-rwxr-xr-x  1 root root     34720 Mar 22  2017 lsns*
-rwxr-xr-x  1 root root    115136 Mar 23  2012 mawk*
-rwxr-xr-x  1 root root     26500 Mar 22  2017 mcookie*
-rwxr-xr-x  1 root root     38872 Feb 22  2017 md5sum*
lrwxrwxrwx  1 root root         6 Feb 22  2017 md5sum.textutils -> md5sum*
-rwxr-xr-x  1 root root      9916 Mar 22  2017 mesg*
-rwxr-xr-x  1 root root     55384 Feb 22  2017 mkfifo*
-rwxr-xr-x  1 root root     26412 Mar 22  2017 namei*
lrwxrwxrwx  1 root root        22 Feb 13 00:00 nawk -> /etc/alternatives/nawk*
-rwsr-xr-x  1 root root     30844 May 17  2017 newgrp*
-rwxr-xr-x  1 root root     30656 Feb 22  2017 nice*
-rwxr-xr-x  1 root root     38944 Feb 22  2017 nl*
-rwxr-xr-x  1 root root     30656 Feb 22  2017 nohup*
-rwxr-xr-x  1 root root     30660 Feb 22  2017 nproc*
-rwxr-xr-x  1 root root     26564 Mar 22  2017 nsenter*
-rwxr-xr-x  1 root root     22388 Nov 24 09:22 nstat*
-rwxr-xr-x  1 root root     59436 Feb 22  2017 numfmt*
-rwxr-xr-x  1 root root     63492 Feb 22  2017 od*
lrwxrwxrwx  1 root root        23 Feb 13 00:00 pager -> /etc/alternatives/pager*
-rwxr-xr-x  1 root root     76144 Mar 22  2017 partx*
-rwsr-xr-x  1 root root     49688 May 17  2017 passwd*
-rwxr-xr-x  1 root root     30660 Feb 22  2017 paste*
-rwxr-xr-x  1 root root     26548 Feb 22  2017 pathchk*
-rwxr-xr-x  2 root root   1816756 Sep 12 16:37 perl*
-rwxr-xr-x  2 root root   1816756 Sep 12 16:37 perl5.24.1*
-rwxr-xr-x  1 root root     42920 Mar 22  2017 pg*
-rwxr-xr-x  1 root root     34828 Feb 22  2017 pinky*
-rwxr-xr-x  1 root root     14056 Jun 15  2017 pldd*
-rwxr-xr-x  1 root root     71764 Feb 22  2017 pr*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 printenv*
-rwxr-xr-x  1 root root     42948 Feb 22  2017 printf*
-rwxr-xr-x  1 root root     30908 Mar 22  2017 prlimit*
-rwxr-xr-x  1 root root     63564 Feb 22  2017 ptx*
-rwxr-xr-x  1 root root     42968 Feb 22  2017 realpath*
-rwxr-xr-x  1 root root      9932 Mar 22  2017 rename.ul*
-rwxr-xr-x  1 root root      9856 Mar 22  2017 renice*
lrwxrwxrwx  1 root root         4 Sep  7 17:05 [ 1082.126171] docker0: port 1(veth656f43a) entered disabled state
reset -> tset*[ 1082.747139] device veth656f43a left promiscuous mode

-rwxr-xr-x  1 root root     34708 Mar 22  2017 resizepart*
-rwxr-xr-x  1 root root      9920 Mar 22  2017 rev*
-rwxr-xr-x  1 root root        30 Nov 28  2016 rgrep*
-rwxr-xr-x  1 root root       173 Nov 24 09:22 routef*
-rwxr-xr-x  1 root root      1262 Nov 24 09:22 routel*
lrwxrwxrwx  1 root root         6 Nov 24 09:22 rtstat -> lnstat*
-rwxr-xr-x  1 root root     30704 Feb 22  2017 runcon*
-rwxr-xr-x  1 root root     10469 Apr  2  2017 savelog*
-rwxr-xr-x  1 root root     22352 Mar 22  2017 [ 1082.757324] docker0: port 1(veth656f43a) entered disabled state
script*
-rwxr-xr-x  1 root root     22304 Mar 22  2017 scriptreplay*
-rwxr-xr-x  1 root root     42928 Jan  9  2017 sdiff*
-rwxr-xr-x  1 root root      1215 Dec 20 13:39 select-editor*
-rwxr-xr-x  1 root root      1138 Dec 20 13:39 sensible-browser*
-rwxr-xr-x  1 root root      1109 Dec 20 13:39 sensible-editor*
-rwxr-xr-x  1 root root       288 Dec 20 13:39 sensible-pager*
-rwxr-xr-x  1 root root     42972 Feb 22  2017 seq*
-rwxr-xr-x  1 root root     14072 Mar 22  2017 setarch*
-rwxr-xr-x  1 root root      9916 Mar 22  2017 setsid*
-rwxr-xr-x  1 root root     38716 Mar 22  2017 setterm*
lrwxrwxrwx  1 root root         6 May 17  2017 sg -> newgrp*
-rwxr-xr-x  1 root root     42968 Feb 22  2017 sha1sum*
-rwxr-xr-x  1 root root     47064 Feb 22  2017 sha224sum*
-rwxr-xr-x  1 root root     47064 Feb 22  2017 sha256sum*
-rwxr-xr-x  1 root root    124888 Feb 22  2017 sha384sum*
-rwxr-xr-x  1 root root    124888 Feb 22  2017 sha512sum*
-rwxr-xr-x  1 root root     55348 Feb 22  2017 shred*
-rwxr-xr-x  1 root root     51216 Feb 22  2017 shuf*
-rwxr-xr-x  1 root root    100684 Feb 22  2017 sort*
-rwxr-xr-x  1 root root     51668 Feb 22  2017 split*
-rwxr-xr-x  1 root root     75892 Feb 22  2017 stat*
-rwxr-xr-x  1 root root     42968 Feb 22  2017 stdbuf*
-rwxr-xr-x  1 root root     38876 Feb 22  2017 sum*
-rwxr-xr-x  1 root root     13768 Sep  7 17:05 tabs*
-rwxr-xr-x  1 root root     34776 Feb 22  2017 tac*
-rwxr-xr-x  1 root root     67636 Feb 22  2017 tail*
-rwxr-xr-x  1 root root     26460 Mar 22  2017 taskset*
-rwxr-xr-x  1 root root     30680 Feb 22  2017 tee*
-rwxr-xr-x  1 root root     42932 Feb 22  2017 test*
-rwxr-xr-x  1 root root     75284 Sep  7 17:05 tic*
-rwxr-xr-x  1 root root     35248 Feb 22  2017 timeout*
-rwxr-xr-x  1 root root     13768 Sep  7 17:05 toe*
lrwxrwxrwx  1 root root        10 Feb 13 00:00 touch -> /bin/touch*
-rwxr-xr-x  1 root root     13796 Sep  7 17:05 tput*
-rwxr-xr-x  1 root root     42960 Feb 22  2017 tr*
-rwxr-xr-x  1 root root     34752 Feb 22  2017 truncate*
-rwxr-xr-x  1 root root     17868 Sep  7 17:05 tset*
-rwxr-xr-x  1 root root     30656 Feb 22  2017 tsort*
-rwxr-xr-x  1 root root     26544 Feb 22  2017 tty*
-rwxr-xr-x  1 root root     15183 Jun 15  2017 tzselect*
-rwxr-xr-x  1 root root     30684 Feb 22  2017 unexpand*
-rwxr-xr-x  1 root root     38880 Feb 22  2017 uniq*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 unlink*
-rwxr-xr-x  1 root root     18292 Mar 22  2017 unshare*
-rwxr-xr-x  1 root root     42448 May 17  2017 update-alternatives*
-rwxr-xr-x  1 root root     26568 Feb 22  2017 users*
-rwxr-xr-x  1 root root     18264 Mar 22  2017 utmpdump*
-rwxr-sr-x  1 root tty      26496 Mar 22  2017 wall*
-rwxr-xr-x  1 root root     43020 Feb 22  2017 wc*
-rwxr-xr-x  1 root root     22512 Mar 22  2017 whereis*
lrwxrwxrwx  1 root root        10 Feb 13 00:00 which -> /bin/which*
-rwxr-xr-x  1 root root     43016 Feb 22  2017 who*
-rwxr-xr-x  1 root root     26548 Feb 22  2017 whoami*
-rwxr-xr-x  1 root root     58956 Feb 18  2017 xargs*
-rwxr-xr-x  1 root root     26540 Feb 22  2017 yes*
-rwxr-xr-x  1 root root     13996 Jun 15  2017 zdump*
 ---> f2e07e847159
Removing intermediate container fa9af5f39152
Step 4/6 : RUN touch /usr/bin/bogus
 ---> Running in cacf8cee10e4
[ 1089.914715] device veth3c146a4 entered promiscuous mode
[ 1091.115031] eth0: renamed from vethb9541f3
[ 1091.126587] docker0: port 1(veth3c146a4) entered forwarding state
[ 1091.138538] docker0: port 1(veth3c146a4) entered forwarding state
[ 1091.779275] vethb9541f3: renamed from eth0
[ 1091.879200] docker0: port 1(veth3c146a4) entered disabled state
[ 1091.892149] device veth3c146a4 left promiscuous mode
[ 1091.901799] docker0: port 1(veth3c146a4) entered disabled state
 ---> 0cfa91f06602
Removing intermediate container cacf8cee10e4
Step 5/6 : RUN ls -laF /usr
 ---> Running in f53f99030638
[ 1101.063642] device vethc262c96 entered promiscuous mode
[ 1102.861391] eth0: renamed from veth608e3c6
[ 1102.871385] docker0: port 1(vethc262c96) entered forwarding state
[ 1102.883329] docker0: port 1(vethc262c96) entered forwarding state
total 12
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ./
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ../
drwxr-xr-x 1 root root 4096 Mar  1 14:26 bin/
[ 1103.515778] veth608e3c6: renamed from eth0
[ 1103.623798] docker0: port 1(vethc262c96) entered disabled state
[ 1103.637259] device vethc262c96 left promiscuous mode
[ 1103.646955] docker0: port 1(vethc262c96) entered disabled state
 ---> 5aa7f83a23f3
Removing intermediate container f53f99030638
Step 6/6 : RUN ls -laF ./usr/bin
 ---> Running in a2e45cdcfe3d
[ 1105.592667] device veth1b312c4 entered promiscuous mode
[ 1106.795150] eth0: renamed from veth5bf72af
[ 1106.805867] docker0: port 1(veth1b312c4) entered forwarding state
[ 1106.817816] docker0: port 1(veth1b312c4) entered forwarding state
total 8
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ./
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ../
-rw-r--r-- 1 root root    0 Mar  1 14:26 bogus
[ 1107.751294] veth5bf72af: renamed from eth0
[ 1107.824235] docker0: port 1(veth1b312c4) entered disabled state
[ 1107.858899] docker0: port 1(veth1b312c4) entered disabled state
[ 1107.871816] device veth1b312c4 left promiscuous mode
[ 1107.881508] docker0: port 1(veth1b312c4) entered disabled state
 ---> 0d8837d7acaa
Removing intermediate container a2e45cdcfe3d
Successfully built 0d8837d7acaa
[ 1110.294006] device veth333ebcb entered promiscuous mode
[ 1111.457522] eth0: renamed from veth38c3f7e
[ 1111.467371] docker0: port 1(veth333ebcb) entered forwarding state
[ 1111.479312] docker0: port 1(veth333ebcb) entered forwarding state
total 12
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ./
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ../
drwxr-xr-x 1 root root 4096 Mar  1 14:26 bin/
total 8
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ./
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ../
-rw-r--r-- 1 root root    0 Mar  1 14:26 bogus
sh: 1: whoami: not found
FAIL
[ 1112.182808] veth38c3f7e: renamed from eth0
[ 1112.310943] docker0: port 1(veth333ebcb) entered disabled state
[ 1112.336909] device veth333ebcb left promiscuous mode
[ 1112.346864] docker0: port 1(veth333ebcb) entered disabled state
total 12
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ./
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ../
drwxr-xr-x 1 root root 4096 Mar  1 14:26 bin/
total 8
drwxr-xr-x 1 root root 4096 Mar  1 14:26 ./
drwxr-xr-x 1 root root 4096 Feb 13 00:00 ../
-rw-r--r-- 1 root root    0 Mar  1 14:26 bogus
sh: 1: whoami: not found
FAIL
test_container
Untagged: test:latest
Deleted: sha256:0d8837d7acaa68aca50198f91aaeabeae21bd8379a0443e15227bb069eee7391
Deleted: sha256:5aa7f83a23f36902b86e29da2e2ea196a8441cdbe9cdecef0e54608d9c2b1234
Deleted: sha256:0cfa91f06602a053ed6407af6704eb327e5d52673d7ccb17bf474db3ce86920b
Deleted: sha256:521252d530f607f3769b5aeb5867e583ccfcd9b275a140d6a7cbf62f20af4838
Deleted: sha256:f2e07e8471597350cb19a6f288e58ac0f788acc361faee0bde63226a328772d9
Deleted: sha256:a8d0ec2e105356c35bdb6c86284677b6f0fe0c43ff40f87331cbddcb9089ea01

And here is the test script:

# cat test.sh
docker build -t test - <<"EOF"
FROM debian
RUN ls -laF /usr
RUN ls -laF ./usr/bin
RUN touch /usr/bin/bogus
RUN ls -laF /usr
RUN ls -laF ./usr/bin
EOF

docker run -it --name=test_container test sh -c "ls -laF /usr ; ls -laF ./usr/bin ; whoami && echo PASS || echo FAIL"

docker logs test_container

docker rm test_container

docker rmi test

sdetweil · March 1, 2018, 2:39pm

ain’t that interesting…

spierepf · March 1, 2018, 2:45pm

Indeed. I think my yocto build is misconfigured somewhere. Possibly, there is a kernel config flag that I need to set, but I don’t know which one, or what to set it to.

spierepf · March 5, 2018, 2:28pm

I’ve added check-config.sh to my host system.

# /usr/share/docker/contrib/check-config.sh 
info: reading kernel config from /proc/config.gz ...

Generally Necessary:
- cgroup hierarchy: properly mounted [/sys/fs/cgroup]
- CONFIG_NAMESPACES: enabled
- CONFIG_NET_NS: enabled
- CONFIG_PID_NS: enabled
- CONFIG_IPC_NS: enabled
- CONFIG_UTS_NS: enabled
- CONFIG_CGROUPS: enabled
- CONFIG_CGROUP_CPUACCT: enabled
- CONFIG_CGROUP_DEVICE: enabled
- CONFIG_CGROUP_FREEZER: enabled
- CONFIG_CGROUP_SCHED: enabled
- CONFIG_CPUSETS: enabled
- CONFIG_MEMCG: enabled
- CONFIG_KEYS: enabled
- CONFIG_VETH: enabled
- CONFIG_BRIDGE: enabled (as module)
- CONFIG_BRIDGE_NETFILTER: enabled (as module)
- CONFIG_NF_NAT_IPV4: enabled (as module)
- CONFIG_IP_NF_FILTER: enabled (as module)
- CONFIG_IP_NF_TARGET_MASQUERADE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_ADDRTYPE: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_CONNTRACK: enabled (as module)
- CONFIG_NETFILTER_XT_MATCH_IPVS: missing
- CONFIG_IP_NF_NAT: enabled (as module)
- CONFIG_NF_NAT: enabled (as module)
- CONFIG_NF_NAT_NEEDED: enabled
- CONFIG_POSIX_MQUEUE: enabled
- CONFIG_DEVPTS_MULTIPLE_INSTANCES: enabled

Optional Features:
- CONFIG_USER_NS: enabled
- CONFIG_SECCOMP: enabled
- CONFIG_CGROUP_PIDS: missing
- CONFIG_MEMCG_SWAP: missing
- CONFIG_MEMCG_SWAP_ENABLED: missing
- CONFIG_MEMCG_KMEM: enabled
- CONFIG_BLK_CGROUP: missing
- CONFIG_BLK_DEV_THROTTLING: missing
- CONFIG_IOSCHED_CFQ: enabled (as module)
- CONFIG_CFQ_GROUP_IOSCHED: missing
- CONFIG_CGROUP_PERF: enabled
- CONFIG_CGROUP_HUGETLB: missing
- CONFIG_NET_CLS_CGROUP: enabled (as module)
- CONFIG_CGROUP_NET_PRIO: enabled
- CONFIG_CFS_BANDWIDTH: enabled
- CONFIG_FAIR_GROUP_SCHED: enabled
- CONFIG_RT_GROUP_SCHED: enabled
- CONFIG_IP_VS: enabled (as module)
- CONFIG_IP_VS_NFCT: enabled
- CONFIG_IP_VS_RR: enabled (as module)
- CONFIG_EXT3_FS: missing
- CONFIG_EXT3_FS_XATTR: missing
- CONFIG_EXT3_FS_POSIX_ACL: missing
- CONFIG_EXT3_FS_SECURITY: missing
    (enable these ext3 configs if you are using ext3 as backing filesystem)
- CONFIG_EXT4_FS: enabled
- CONFIG_EXT4_FS_POSIX_ACL: enabled
- CONFIG_EXT4_FS_SECURITY: enabled
- Network Drivers:
  - "overlay":
    - CONFIG_VXLAN: missing
      Optional (for encrypted networks):
      - CONFIG_CRYPTO: enabled
      - CONFIG_CRYPTO_AEAD: missing
      - CONFIG_CRYPTO_GCM: missing
      - CONFIG_CRYPTO_SEQIV: missing
      - CONFIG_CRYPTO_GHASH: missing
      - CONFIG_XFRM: enabled
      - CONFIG_XFRM_USER: enabled (as module)
      - CONFIG_XFRM_ALGO: enabled (as module)
      - CONFIG_INET_ESP: missing
      - CONFIG_INET_XFRM_MODE_TRANSPORT: enabled
  - "ipvlan":
    - CONFIG_IPVLAN: missing
  - "macvlan":
    - CONFIG_MACVLAN: enabled
    - CONFIG_DUMMY: missing
  - "ftp,tftp client in container":
    - CONFIG_NF_NAT_FTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_FTP: enabled (as module)
    - CONFIG_NF_NAT_TFTP: enabled (as module)
    - CONFIG_NF_CONNTRACK_TFTP: enabled (as module)
- Storage Drivers:
  - "aufs":
    - CONFIG_AUFS_FS: missing
  - "btrfs":
    - CONFIG_BTRFS_FS: missing
    - CONFIG_BTRFS_FS_POSIX_ACL: missing
  - "devicemapper":
    - CONFIG_BLK_DEV_DM: missing
    - CONFIG_DM_THIN_PROVISIONING: missing
  - "overlay":
    - CONFIG_OVERLAY_FS: enabled (as module)
  - "zfs":
    - /dev/zfs: missing
    - zfs command: missing
    - zpool command: missing

Limits:
- /proc/sys/kernel/keys/root_maxkeys: 1000000