Runing hello-world leads to "stream copy error: reading from a closed fifo" on Alpine Linux (RPI)

Hello!
I got “stream copy error: reading from a closed fifo” error message in /var/log/docker.log when I run simple container hello-world. :frowning:
I have raspberry pi 2 with Alpine Linux installed on it.
I run docker under root account or with sudo.

sudo docker run hello-world

docker: Error response from daemon: OCI runtime create failed: container_linux.go:344: starting container process caused “process_linux.go:297: getting the final child’s pid from pipe caused "EOF"”: unknown.

I run docker-18.09.1 under stable Alpine Linux release. The same error happens when I run docker-18.09.3 version under latest develepment Apline Linux release (edge).

Here is docker info:

Containers: 3
Running: 0
Paused: 0
Stopped: 3
Images: 1
Server Version: 18.09.1-ce
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9754871865f7fe2f4e74d43e2fc7ccd237edcbce
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: v0.18.0 (expected: fec3683b971d9c3ef73f284f176672c44b448662)
Kernel Version: 4.19.26-0-rpi2
Operating System: Alpine Linux v3.9
OSType: linux
Architecture: armv7l
CPUs: 4
Total Memory: 927.1MiB
Name: tatyana2
ID: 2YDL:LOYA:2XIG:CJZV:WJZ3:EI43:Z5UY:JZVE:DSI7:BYUG:DIWZ:RECZ
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support

docker version

Client:
Version: 18.09.1-ce
API version: 1.39
Go version: go1.11.4
Git commit: 4c52b901c6cb019f7552cd93055f9688c6538be4
Built: Fri Jan 11 15:41:50 2019
OS/Arch: linux/arm
Experimental: false

Server:
Engine:
Version: 18.09.1-ce
API version: 1.39 (minimum version 1.12)
Go version: go1.11.4
Git commit: 4c52b901c6cb019f7552cd93055f9688c6538be4
Built: Fri Jan 11 15:40:54 2019
OS/Arch: linux/arm
Experimental: false

a part of log with errors:

time=“2019-03-17T10:46:29.130081557+03:00” level=info msg=“shim containerd-shim started” address=“/containerd-shim/moby/537fe49280c5f2c3d6f857b33e69026905e114c5d5c745db58ffe1f3f231e008/shim.sock” debug=false pid=1641
time=“2019-03-17T10:46:29.866022121+03:00” level=info msg=“shim reaped” id=537fe49280c5f2c3d6f857b33e69026905e114c5d5c745db58ffe1f3f231e008
> time=“2019-03-17T10:46:29.875501070+03:00” level=error msg=“stream copy error: reading from a closed fifo”
> time=“2019-03-17T10:46:29.883295012+03:00” level=error msg=“stream copy error: reading from a closed fifo”
time=“2019-03-17T10:46:33.519631331+03:00” level=error msg=“537fe49280c5f2c3d6f857b33e69026905e114c5d5c745db58ffe1f3f231e008 cleanup: failed to delete container from containerd: no such container”
time=“2019-03-17T10:46:33.520243781+03:00” level=error msg=“Handler for POST /v1.39/containers/537fe49280c5f2c3d6f857b33e69026905e114c5d5c745db58ffe1f3f231e008/start returned error: OCI runtime create failed: container_linux.go:344: starting container process caused "process_linux.go:297: getting the final child’s pid from pipe caused \"EOF\"": unknown”

full logs is attached docker.log (12.3 KB)

at least tell me please direction of digging

I am having this same problem on my Raspberry Pi 3 with armv7 Alpine Linux 3.9.2. Docker version is 18.09.3-ce. I also have tried the steps on this page https://wiki.alpinelinux.org/wiki/Docker section " “WARNING: No {swap,memory} limit support.”

I run this command and get the same error:

$ docker run -it --rm arm32v7/alpine:3.9.2 /bin/ash

Hello, fa11risk!

I tried the same section “WARNING: No {swap,memory} limit support.” too. But it did not help.

By the way how did you install latest docker 18.09.3 for Alpine Linux 3.9.2? Did you use binaries from here https://download.docker.com/linux/static/stable/ ?

PS. And if it helps I tried install docker on RPi 2.

I have installed latest docker from here https://download.docker.com/linux/static/stable/armhf/
and it works! :slight_smile:
at least hello world. But anyway I got some progress. Thank you for the hint.
I use docker-18.09.3.tgz
What about you?

PS. do not forget install iptables

I got my docker from Alpine Linux “edge” package source. I wonder if the one you downloaded is different than what is in edge.

Good question. May be they become different at compiling, for example, flags, compiler or somtheing else. But the fact is binaries work from official docker repository but they do not work from alpine repository.
To make it work I just download from https://download.docker.com/linux/static/stable/armhf/ and replace all binaries placing in /usr/bin by binaries of the downloaded archive. That is it.
I know better to keep binaries from docker repository in /usr/local/bin. But for experiment everthing is possible.

Look. I have tried to use binaries from docker repository version 18.9.1 and even they work. It is magic. I do not know how to explain it.

What I have found out for now.
(1).To get working docker you need to replace only runc. Get it from here Index of linux/static/stable/armhf/ and put it to /usr/bin/runc.
(2).Some fork inside of apline runc (from the distributive) got SIGSEGV so it does not work.
I use strace to get deeper:

6095 execve(“/usr/bin/runc.apk.exec”, [“/usr/bin/runc.apk.exec”, “–root”, “/var/run/docker/runtime-runc/moby”, “–log”, “/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ff7e11de778883b511cb6c19e1b543b2b6814e97763d4119a97ca8ab2bcfbcf5/log.json”, “–log-format”, “json”, “create”, “–bundle”, “/var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ff7e11de778883b511cb6c19e1b543b2b6814e97763d4119a97ca8ab2bcfbcf5”, “–pid-file”, “/var/run/docker/containerd/daemon/io.containerd.runtime.v1.linux/moby/ff7e11de778883b511cb6c19e1b543b2b6814e97763d4119a97ca8ab2bcfbcf5/init.pid”, “ff7e11de778883b511cb6c19e1b543b2b6814e97763d4119a97ca8ab2bcfbcf5”], 0x7eecca48 /* 21 vars /) = 0
6095 clone(child_stack=0x66cedd60, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x66cedd7c, tls=0x66cedde0, child_tidptr=0x66cedd84) = 6096
6095 clone(child_stack=0x66cd6d60, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x66cd6d7c, tls=0x66cd6de0, child_tidptr=0x66cd6d84) = 6097
6095 clone(child_stack=0x66cbfd60, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x66cbfd7c, tls=0x66cbfde0, child_tidptr=0x66cbfd84) = 6098
6095 clone(child_stack=0x66ca8d60, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x66ca8d7c, tls=0x66ca8de0, child_tidptr=0x66ca8d84) = 6099
6099 clone(child_stack=0x66c51d60, flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID|0x400000, parent_tidptr=0x66c51d7c, tls=0x66c51de0, child_tidptr=0x66c51d84) = 6100
6100 clone(child_stack=NULL, flags=SIGCHLD) = 6101
6101 execve(“/proc/self/exe”, [“/usr/bin/runc.apk.exec”, “init”], 0x184e500 /
4 vars /) = 0
6101 execve(“/proc/self/fd/5”, [“/usr/bin/runc.apk.exec”, “init”], 0xad7f80 /
4 vars */) = 0
6101 clone(child_stack=0x7ee04920, flags=CLONE_PARENT|SIGCHLD) = 6102
6102 — SIGSEGV {si_signo=SIGSEGV, si_code=SEGV_MAPERR, si_addr=0x2c} —
6102 +++ killed by SIGSEGV +++
6100 — SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=6102, si_uid=0, si_status=SIGSEGV, si_utime=0, si_stime=0} —
6101 — SIGPIPE {si_signo=SIGPIPE, si_code=SI_USER, si_pid=6101, si_uid=0} —
6100 waitid(P_PID, 6101, <unfinished …>
6101 +++ killed by SIGPIPE +++
6100 <… waitid resumed> {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=6101, si_uid=0, si_status=SIGPIPE, si_utime=0, si_stime=0}, WEXITED|WNOWAIT, NULL) = 0
6100 — SIGCHLD {si_signo=SIGCHLD, si_code=CLD_KILLED, si_pid=6101, si_uid=0, si_status=SIGPIPE, si_utime=4, si_stime=6} —
6100 wait4(6101, [{WIFSIGNALED(s) && WTERMSIG(s) == SIGPIPE}], 0, {ru_utime={tv_sec=0, tv_usec=37141}, ru_stime={tv_sec=0, tv_usec=55712}, …}) = 6101
6100 exit_group(1) = ?
6098 +++ exited with 1 +++
6100 +++ exited with 1 +++
6097 +++ exited with 1 +++
6096 +++ exited with 1 +++
6099 +++ exited with 1 +++
6095 +++ exited with 1 +++

(3).When I use runc from docker site then no SIGSEGV and SIGPIPE only zero exit codes you can see in strace logs.
When run runc from apk (alpine compiled) without arguments you got version of it. In version you can find commit. Commit may be from which runc built was.
So, alpine apk runc and runc from docker site has different commits specified in help.
For alpine it is 6635b4f0c6af3810594d2770f662f34ddc15b40d.
And for original docker runc is 96ec2177ae841256168fcf76954f7177af9446eb.
There is a need to see differences between these commits. But I think I will do it next time.

If I am right then runc built from here GitHub - opencontainers/runc: CLI tool for spawning and running containers according to the OCI specification

I loaded aarch64 alpine onto my Raspberry Pi 3. Then I added community 3.9 (did not enable edge sources) and did an apk update followed by apk upgrade. Then I installed docker. Docker worked fine this time. I ran the following:

$ docker run --rm -it arm64v8/alpine:3.9.2 /bin/ash

Great! But seems not for me. :frowning:
Can you please tell me output of command runc 2>&1 | grep commit?
And tell me mirrors/repositories you use too please

I wrote about this problem in a ticket https://bugs.alpinelinux.org/issues/10139
So follow it if you face the same problem