Message: stat /sys/fs/<container_id>: no such file or directory

Expected behavior

Container should start

Actual behavior

Error message is thrown, container is not started. Works with Toolbox 1.11-rc2 as well as all previous versions.

Information

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.10.3-beta5
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      lofs
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160331-104041.tar.gz.
Your unique id in bugsnag is: BE33CC85-B017-493C-A01F-678488F21019
Please quote this in all correspondence.

Steps to reproduce the behavior

Given a shell script:

> cat reproduce.sh 
docker run \
  --pid="host" --privileged="true" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /

docker create \
  --volume="/:/rootfs:ro" \
  --volume="/sys:/sys:ro" \
  --volume="/dev:/dev" \
  --volume="/var/run:/var/run:rw" \
  --volume="/var/lib/docker/:/var/lib/docker:rw" \
  --volume="/var/lib/kubelet:/var/lib/kubelet:rw,rshared" \
  --volume="/var/run/docker.sock:/docker.sock" \
  --name="kubelet-volumes" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      true

docker run -d \
  --name="kubelet" \
  --privileged="true" --net="host" --pid="host" \
  --volumes-from="kubelet-volumes" \
  --restart="always" \
    weaveworks/kubernetes-anywhere:kubelet-v1.2

sleep 10

docker rm -f $(docker ps -qa)

docker run \
  --pid="host" --privileged="true" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /

docker create \
  --volume="/:/rootfs:ro" \
  --volume="/sys:/sys:ro" \
  --volume="/dev:/dev" \
  --volume="/var/run:/var/run:rw" \
  --volume="/var/lib/docker/:/var/lib/docker:rw" \
  --volume="/var/lib/kubelet:/var/lib/kubelet:rw,rshared" \
  --volume="/var/run/docker.sock:/docker.sock" \
  --name="kubelet-volumes" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      true

docker run -d \
  --name="kubelet" \
  --privileged="true" --net="host" --pid="host" \
  --volumes-from="kubelet-volumes" \
  --restart="always" \
    weaveworks/kubernetes-anywhere:kubelet-v1.2

And a freshly booted Docker for Mac VM:

> docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Run the script:

> sh -x reproduce.sh 
+ docker run --pid=host --privileged=true weaveworks/kubernetes-anywhere:toolbox-v1.2 nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /
+ docker create --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rshared --volume=/var/run/docker.sock:/docker.sock --name=kubelet-volumes weaveworks/kubernetes-anywhere:toolbox-v1.2 true
9be553dd219bf739a152b9dd88200a7447794684c81d0d6745c5b0ffb2112c4f
+ docker run -d --name=kubelet --privileged=true --net=host --pid=host --volumes-from=kubelet-volumes --restart=always weaveworks/kubernetes-anywhere:kubelet-v1.2
85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b
+ sleep 10
++ docker ps -qa
+ docker rm -f 85d238640224 9be553dd219b 4aa23a67e448
9be553dd219b
4aa23a67e448
Failed to remove container (85d238640224): Error response from daemon: Unable to remove filesystem for 85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b: remove /var/lib/docker/containers/85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b/shm: device or resource busy
+ docker run --pid=host --privileged=true weaveworks/kubernetes-anywhere:toolbox-v1.2 nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /
Timestamp: 2016-03-31 09:35:24.824666564 +0000 UTC
Code: System error

Message: stat /sys/fs/91023a3855110b7a74ddf0cf1fb5cc70b2cc59fc3c4f8523cc12b8d483d2a016: no such file or directory

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@111
---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Container command not found or does not exist..
+ docker create --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rshared --volume=/var/run/docker.sock:/docker.sock --name=kubelet-volumes weaveworks/kubernetes-anywhere:toolbox-v1.2 true
1c6dff4e68fa1567a6e5dd197a0a0d3cb129ddf8b8d4d27fc0b6f634c366046d
+ docker run -d --name=kubelet --privileged=true --net=host --pid=host --volumes-from=kubelet-volumes --restart=always weaveworks/kubernetes-anywhere:kubelet-v1.2
aa3b24985db6f435b0cde39518715e324be9a0cd6f38232e5291d3a6c5f8c553
docker: Error response from daemon: Container command not found or does not exist..

Now the daemon fails to start any container whatsoever:

> docker run -ti alpine sh -l
Timestamp: 2016-03-31 09:47:24.655733308 +0000 UTC
Code: System error

Message: stat /sys/fs/007506cc4b07bc58cbaf6a9021b7ce216122abd0a2f5a474c9d8d5aa1b4c964f: no such file or directory

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@111
---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Container command not found or does not exist..

Added to internal issue tracker as #2161

I’ve re-tested using vanilla 1.10.3 binary on Debian 8 and the same script runs just fine…

I can’t actually reproduce this on the current master version, will retest on the last beta as well as I don’t know of any related changes.

Can you reproduce if you do a Settings / reset to factory defaults first?

Yes, I am able to reproduce it even after resetting to factory defaults.

Odd I also can’t reproduce on a fresh beta install on my osx VM.

Is there anything in the logs that looks odd?

Ah yes I can reproduce on the beta, I had to run another container after running the script. The master build seems ok though, even though I cannot immediately see which change might have fixed this. Will do some more testing but hopefully this is fixed in next week’s release.

1 Like

Not much that I can see, but please take [quote=“justincormack, post:7, topic:8109”]
I cannot immediately see which change might have fixed this.
[/quote]

Some issue in Docker repository suggest that errors like this could be related to the storage driver, has that changed?

No there are no storage driver changes since the beta.

Will master make it to the release channel any time soon? I’d be happy to play around with snapshot builds :wink:

There will be a new beta on Tuesday.

2 Likes

@errordeveloper Can you confirm that the latest release solves your issue?

@frenchben @justincormack yes, I confirm that I’m not able to reproduce this issue with latest version.