Simple hello.c compiled runs on Mac, but not in container

Docker Desktop for Mac
1

Using Mac OSX (Sierra), I compiled a simple hello.c file (similar to the one on the GitHub repo). The executable runs fine from the Mac command line, but fails to run from within the container; I get the following error, "standard_init_linux.go:175: exec user process caused “exec format error”. What am I doing wrong?

(I understand that my little “hello” application could be built based on a full on Linux or Debian OS base image and probably work, but I’m trying to understand what’s the absolute minimal version of an application that I can build and as part of that process, I want to understand what the minimum environment for the Docker image to contain. If my hello.c application makes native Linux/Debian OS kernel calls, e.g write() and _exit, I would have assumed that it could execute this without any issues. In other words, I was assuming that Docker would be able to make this call directly to the host OS, in this case Debian. However, I tried the same exact thing using my Linux host OS and it fails with another error: panic: standard_init_linux.go:175: exec user process caused "no such file or directory" [recovered] )

— hello.c —

#include <unistd.h>  // _exit() and write(2); man -s2 write

#ifndef DOCKER_IMAGE
        #define DOCKER_IMAGE "hello-world"
#endif

#ifndef DOCKER_GREETING
        #define DOCKER_GREETING "Hello from Docker!"
#endif

const char message[] =
        "\n"
        DOCKER_GREETING "\n"
        "This message shows that your C application appears to be working correctly.\n"
        "\n";

int main() {
        write(1, message, sizeof(message) - 1);
        _exit(0);
}

I compile it on Mac OSX with:
build -o hello hello.c

$ ./hello

Hello from Docker!
This message shows that your C application appears to be working correctly.

I then build the Docker image with a minimal.docker file that looks like:
$ cat minimal.docker
FROM scratch
ADD hello /
CMD ["/hello"]

$ docker build -f minimal.docker .
Sending build context to Docker daemon 13.82 kB
Step 1/3 : FROM scratch
—>
Step 2/3 : ADD hello /
—> 56e31a943dde
Removing intermediate container 843df5833844
Step 3/3 : CMD /hello
—> Running in 1cfc46e8e0c4
—> 21c49c4ad4bf
Removing intermediate container 1cfc46e8e0c4
Successfully built 21c49c4ad4bf

Then I try to execute it with:
$ docker run 21c49c4ad4bf

And I get the following error:
standard_init_linux.go:175: exec user process caused “exec format error”

I’ve read things that suggest that I need to verify that I am trying to run a 64-bit executable with my 64-bit Mac OSX Sierra OS. And I am as can be seen with:

$ file hello
hello: Mach-O 64-bit executable x86_64

Here’s my Docker version:

$ docker version
Client:
 Version:      1.13.0-rc4
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   88862e7
 Built:        Sat Dec 17 01:34:17 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      1.13.0-rc4
 API version:  1.25 (minimum version 1.12)
 Go version:   go1.7.3
 Git commit:   88862e7
 Built:        Sat Dec 17 01:34:17 2016
 OS/Arch:      linux/amd64
 Experimental: true

What am I doing wrong? I was expecting the Docker container to send the calls directly to the Mac OSX Debian kernel to execute.

2

You’re trying to run a Mac binary on Linux. (Docker will run native Linux binaries, independent of the host OS; there is an intermediate hidden Linux VM.)

As an additional note, if you’re bundling a precompiled binary into a FROM scratch image, you need to make sure it’s a static binary. (gcc by default will produce a dynamic binary, and so you need at least /lib/ld-linux.so.2 and whatever libraries it needs to run it.)

3

Thanks for the quick reply. You commented that I was “trying to run a Mac binary on Linux”, but I compiled the hello.c file on my Mac and I’m trying to run it via Docker for Mac. But, as I type this, I now remember that Docker for Mac is running a Linux VM. So, the Linux VM must be intercepting the Debian OS call (write() and _exit() ) that I thought I was making directly to the Host OS (Debian). Compiling a Linux binary on MacOSX would require an extensive toolchain to support and this is more than my little experiment is worth doing; but I already have a Linux box that I can use to build this, so keep reading…

Thanks for the additional note regarding needing to statically link in ld-linux.so.2, to produce a static binary, I was assuming that it would be able to use the VM’s /lib/ld-linux.so.2, but now that I think about it (namespaces, cgroups, etc), that doesn’t make sense either. I’m not sure which one of these would prevent this, but I’m guessing one of them does.

I confirmed that the gcc command that I used (which on Mac OS, is actually using Clang these days) indeed created a dynamic library. For those that stumble on to this question later, you can verify this by running: ldd hello and it will show you what dynamic libraries the executable is dependent on. If it’s not dependent on any dynamic libraries, it will say ‘not a dynamic executable’. Otherwise, it will list the dynamic libraries that it is dependent on.

However, to create a statically linked binary using gcc, you only need to add the flag ‘-static’ to the C compile command, so it looks like this: gcc -o hello -static hello.c. Issuing ldd hello should now show ‘not a dynamic executable’

Another interesting note is that the statically linked library is now 877,127 Bytes; whereas the hello.c file linked dynamically was only 8,583 Bytes. This all makes sense of course, but it’s interesting to note the size difference.

After linking the simple compiled hello.c statically on my Linux OS (Mint), I was able to build the Docker image, and run the Linux version of the executable using Docker. Next, I was able to run the Linux compiled image on my Docker for Mac after figuring out how to move the image to my Mac OS without putting it in a repo. I used this SO tip to figure out how to do that: http://stackoverflow.com/questions/23935141/how-to-copy-docker-images-from-one-host-to-another-without-via-repository

It basically amounted to doing the following:

On Linux box:
$ sudo docker save -o linuxHello

On Mac OSX:
cp or scp linuxHello to macOS
$ docker load -i linuxHello
$ docker run

Hello from Docker!

Thanks David for your tips!

4

Not really:

host$ docker run --rm -it -v $PWD:/build ubuntu:16.04
container# apt-get install build-essential
container# cd /build
container# gcc -o hello hello.c

For simple programs, it may work to just add -static to that gcc command line. (IME this is much more common for Go programs, which have tools like go-bindata to bundle application data into the binary, than for C/C++ programs, which frequently depend on things being in the host filesystem somewhere.)

It’s probably worth remembering the layers you have here:

(1) You have your MacOS host, running an Apple-provided kernel and BSDish userspace. Nothing ever uses anything from this, with the two exceptions that (a) many cases of docker run -v /host/path:/container/path will use real host paths, and (b) outbound network connections ultimately go through the host’s network.

(2) Docker for Mac provides a minimal Linux VM. All system calls go through its kernel. Its filesystem is never used for anything.

(3) Containers provide a complete userspace.

In your example of write(), it is handled first by the /lib/libc.so.6 in the container itself; that will ultimately make a system call, which goes to the VM’s kernel (which is shared across all containers). “Host OS (Debian)” doesn’t make sense: each word is from a different layer in this.

5

The binary created is in a different format. You can’t build something on a mac, and run it in docker (linux) and expect it to run, even something as simple as a hello world.

You would have to build the binary inside the docker container before you can execute it. I’m sure someone else has run into this, but I"m pretty sure you can’t just do g++ on your mac and copy the output and expect it to work fine.

6

This has been sorted per the above discussion. Read through it to see how it was done. Thanks