Docker on armel, what device is missing? [Resolved: Docker is unfit for purpose]

General
1

Hi all,

I’m in the process of trying out Docker on an industrial embedded computer that runs a ARMv5 CPU. I see that Debian Jessie ships a docker.io package for armel in their “backports” repository and I have just ported Linux kernel 4.9.26 to the device and partitioned the SD card with btrfs.

For the curious, the machine is one of these: https://wiki.embeddedarm.com/wiki/TS-7670
We build our own U-Boot and kernel images rather than using the stock 2.6 kernel they ship with. I’ll probably throw the binaries up on https://bne.vrt.com.au/technologicsys/ in a moment.

Note that this is ARMv5 not ARMv6 or v7, armhf will not work. Moving to armhf would be nice, if someone knows of a sub-AU$150 armhf machine that is certified to work at 85°C, I’m all ears, but for now, this is what we have to work with.

So far so good. I found I had to enable a number of kernel options, but eventually, got Docker daemon to start. However, it won’t run an image:

root@ts7670:~# docker info
Containers: 0
Images: 0
Storage Driver: btrfs
 Build Version: Btrfs v3.17
 Library Version: 101
Execution Driver: native-0.2
Kernel Version: 4.9.26-ts7670-20170504-00031-g9226da7
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 109 MiB
Name: ts7670
ID: JJC6:POS3:3YZG:N7VT:TZMC:YMEJ:SC5K:N5MT:A6VA:J6VJ:6P3O:23UK
root@ts7670:~# docker run --rm -ti armel/debian /bin/bash
[ 2822.312277] docker0: port 1(veth6e9fe58) entered blocking state
[ 2822.318467] docker0: port 1(veth6e9fe58) entered disabled state
[ 2822.530803] device veth6e9fe58 entered promiscuous mode
[ 2822.605129] IPv6: ADDRCONF(NETDEV_UP): veth6e9fe58: link is not ready
[ 2824.411578] eth0: renamed from vetha42adde
[ 2824.562378] IPv6: ADDRCONF(NETDEV_CHANGE): veth6e9fe58: link becomes ready
[ 2824.570315] docker0: port 1(veth6e9fe58) entered blocking state
[ 2824.576484] docker0: port 1(veth6e9fe58) entered forwarding state
Timestamp: [ 2825.559813] docker0: port 1(veth6e9fe58) entered disabled state
[ 2825.661789] device veth6e9fe58 left promiscuous mode
[ 2825.667049] docker0: port 1(veth6e9fe58) entered disabled state
FATA[0020] Error response from daemon: Cannot start container 6b728b7f4dbaf49ee894e18e80d90d4bd45fcec65a2f5627de25a9e6837258eb: [8] System error: no such device

Now, clearly I’ve missed something… there is a “device” missing somewhere. Question is, which device?

As an experiment, I’m trying out Debian Sid now to see if its newer docker.io binary works, but if anyone has seen this message and knows how to fix it, I’m all ears.

2

Well, trying Debian Sid at this time… the docker.io package there is completely broken in that it conflicts with the versions of runc and containerd also available on Sid.

Maybe I might try that a little later once Debian ship a newer docker.io… in the meantime I’ve grabbed a Gentoo stage3 tarball and unpacked that as the root, I can experiment with building Docker easier that way. That will at least tell me if Docker can work on this device.

3

Well, after a marathon compiling run spanning weeks (because people kept tripping the power and other interruptions)… I managed to get Docker running:

bash-4.3# docker info
Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 0
Server Version: 17.05.0-ce-rc3
Storage Driver: btrfs
 Build Version: Btrfs v4.6.1
 Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: v0.2.7 (expected: 9048e5e50717ea4497b757314bad98ea3763c145)
runc version: 9c2d8d1 (expected: 9c2d8d184e5da67c95d601382adf14862e4f2228)
init version: v0.14.0 (expected: 949e6facb77383876aeff8a6944dde66b3089574)
Kernel Version: 4.9.26-ts7670-20170504-00032-gcea2562
Operating System: Linux
OSType: linux
Architecture: armv5tejl
CPUs: 1
Total Memory: 109MiB
Name: ts7670
ID: TIPQ:HYTZ:7CR2:5SX2:7BOA:VBZ2:G66J:SAQM:ZJP2:G7UX:PDHP:K5SD
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 13
 Goroutines: 21
 System Time: 2017-05-23T09:15:58.168991328+10:00
 EventsListeners: 0
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

So far, so good. Now to try and run a container. I am just extracting the armel/debian container now… I’ll report back when it is done.

4

First observation so far:

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND                                                         
21689 root      20   0  924332  29792   7072 R 40.4 26.7   7:43.09 /usr/bin/dockerd -p /run/docker.pid

Yep, that’s Docker chewing up over ¼ of the RAM, so about 32MB. This is getting into “unfit for purpose” territory.

… and it fails again:

bash-4.3# docker run --rm -ti armel/debian /bin/bash
Unable to find image 'armel/debian:latest' locally
latest: Pulling from armel/debian
621afcabaaca: Pull complete
Digest: sha256:2fb14a006907e57a9701c242e3f31141e44b31318524184c1bde92c5afe374a9
Status: Downloaded newer image for armel/debian:latest
docker: Error response from daemon: failed to create endpoint xenodochial_goldberg on network bridge: failed to add the host (vethde90387) <=> sandbox (veth103c820) pair interfaces: operation not supported.
ERRO[1451] error getting events from daemon: context canceled
5 
bash-4.3# docker run --rm -ti armel/debian /bin/bash
docker: Error response from daemon: oci runtime error: container_linux.go:247: starting container process caused "process_linux.go:270: sending config to init process caused \"write parent: broken pipe\"".
ERRO[0152] error getting events from daemon: context canceled

Definitely unfit for purpose. Looks like we won’t be bothering with Docker at the device level. The concepts looked good, but it just doesn’t live up to expectations. The overheads are too high and it introduces too many complexities for it to be practical as a means of managing software on an embedded device.