Docker Community Forums

Share and learn in the Docker community.

Message: stat /sys/fs/<container_id>: no such file or directory


(Errordeveloper) #1

Expected behavior

Container should start

Actual behavior

Error message is thrown, container is not started. Works with Toolbox 1.11-rc2 as well as all previous versions.

Information

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.10.3-beta5
Running diagnostic tests:
[OK]      docker-cli
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      lofs
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160331-104041.tar.gz.
Your unique id in bugsnag is: BE33CC85-B017-493C-A01F-678488F21019
Please quote this in all correspondence.

Steps to reproduce the behavior

Given a shell script:

> cat reproduce.sh 
docker run \
  --pid="host" --privileged="true" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /

docker create \
  --volume="/:/rootfs:ro" \
  --volume="/sys:/sys:ro" \
  --volume="/dev:/dev" \
  --volume="/var/run:/var/run:rw" \
  --volume="/var/lib/docker/:/var/lib/docker:rw" \
  --volume="/var/lib/kubelet:/var/lib/kubelet:rw,rshared" \
  --volume="/var/run/docker.sock:/docker.sock" \
  --name="kubelet-volumes" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      true

docker run -d \
  --name="kubelet" \
  --privileged="true" --net="host" --pid="host" \
  --volumes-from="kubelet-volumes" \
  --restart="always" \
    weaveworks/kubernetes-anywhere:kubelet-v1.2

sleep 10

docker rm -f $(docker ps -qa)

docker run \
  --pid="host" --privileged="true" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /

docker create \
  --volume="/:/rootfs:ro" \
  --volume="/sys:/sys:ro" \
  --volume="/dev:/dev" \
  --volume="/var/run:/var/run:rw" \
  --volume="/var/lib/docker/:/var/lib/docker:rw" \
  --volume="/var/lib/kubelet:/var/lib/kubelet:rw,rshared" \
  --volume="/var/run/docker.sock:/docker.sock" \
  --name="kubelet-volumes" \
    weaveworks/kubernetes-anywhere:toolbox-v1.2 \
      true

docker run -d \
  --name="kubelet" \
  --privileged="true" --net="host" --pid="host" \
  --volumes-from="kubelet-volumes" \
  --restart="always" \
    weaveworks/kubernetes-anywhere:kubelet-v1.2

And a freshly booted Docker for Mac VM:

> docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

Run the script:

> sh -x reproduce.sh 
+ docker run --pid=host --privileged=true weaveworks/kubernetes-anywhere:toolbox-v1.2 nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /
+ docker create --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rshared --volume=/var/run/docker.sock:/docker.sock --name=kubelet-volumes weaveworks/kubernetes-anywhere:toolbox-v1.2 true
9be553dd219bf739a152b9dd88200a7447794684c81d0d6745c5b0ffb2112c4f
+ docker run -d --name=kubelet --privileged=true --net=host --pid=host --volumes-from=kubelet-volumes --restart=always weaveworks/kubernetes-anywhere:kubelet-v1.2
85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b
+ sleep 10
++ docker ps -qa
+ docker rm -f 85d238640224 9be553dd219b 4aa23a67e448
9be553dd219b
4aa23a67e448
Failed to remove container (85d238640224): Error response from daemon: Unable to remove filesystem for 85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b: remove /var/lib/docker/containers/85d2386402246d18ab6b7737571d967638b68684b8e4f8021971af6c7662c12b/shm: device or resource busy
+ docker run --pid=host --privileged=true weaveworks/kubernetes-anywhere:toolbox-v1.2 nsenter --mount=/proc/1/ns/mnt -- mount --make-rshared /
Timestamp: 2016-03-31 09:35:24.824666564 +0000 UTC
Code: System error

Message: stat /sys/fs/91023a3855110b7a74ddf0cf1fb5cc70b2cc59fc3c4f8523cc12b8d483d2a016: no such file or directory

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@111
---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Container command not found or does not exist..
+ docker create --volume=/:/rootfs:ro --volume=/sys:/sys:ro --volume=/dev:/dev --volume=/var/run:/var/run:rw --volume=/var/lib/docker/:/var/lib/docker:rw --volume=/var/lib/kubelet:/var/lib/kubelet:rw,rshared --volume=/var/run/docker.sock:/docker.sock --name=kubelet-volumes weaveworks/kubernetes-anywhere:toolbox-v1.2 true
1c6dff4e68fa1567a6e5dd197a0a0d3cb129ddf8b8d4d27fc0b6f634c366046d
+ docker run -d --name=kubelet --privileged=true --net=host --pid=host --volumes-from=kubelet-volumes --restart=always weaveworks/kubernetes-anywhere:kubelet-v1.2
aa3b24985db6f435b0cde39518715e324be9a0cd6f38232e5291d3a6c5f8c553
docker: Error response from daemon: Container command not found or does not exist..

Now the daemon fails to start any container whatsoever:

> docker run -ti alpine sh -l
Timestamp: 2016-03-31 09:47:24.655733308 +0000 UTC
Code: System error

Message: stat /sys/fs/007506cc4b07bc58cbaf6a9021b7ce216122abd0a2f5a474c9d8d5aa1b4c964f: no such file or directory

Frames:
---
0: setupRootfs
Package: github.com/opencontainers/runc/libcontainer
File: rootfs_linux.go@40
---
1: Init
Package: github.com/opencontainers/runc/libcontainer.(*linuxStandardInit)
File: standard_init_linux.go@57
---
2: StartInitialization
Package: github.com/opencontainers/runc/libcontainer.(*LinuxFactory)
File: factory_linux.go@240
---
3: initializer
Package: github.com/docker/docker/daemon/execdriver/native
File: init.go@35
---
4: Init
Package: github.com/docker/docker/pkg/reexec
File: reexec.go@26
---
5: main
Package: main
File: docker.go@18
---
6: main
Package: runtime
File: proc.go@111
---
7: goexit
Package: runtime
File: asm_amd64.s@1721
docker: Error response from daemon: Container command not found or does not exist..

(Justin Cormack) #2

Added to internal issue tracker as #2161


(Errordeveloper) #3

I’ve re-tested using vanilla 1.10.3 binary on Debian 8 and the same script runs just fine…


(Justin Cormack) #4

I can’t actually reproduce this on the current master version, will retest on the last beta as well as I don’t know of any related changes.

Can you reproduce if you do a Settings / reset to factory defaults first?


(Errordeveloper) #5

Yes, I am able to reproduce it even after resetting to factory defaults.


(Justin Cormack) #6

Odd I also can’t reproduce on a fresh beta install on my osx VM.

Is there anything in the logs that looks odd?


(Justin Cormack) #7

Ah yes I can reproduce on the beta, I had to run another container after running the script. The master build seems ok though, even though I cannot immediately see which change might have fixed this. Will do some more testing but hopefully this is fixed in next week’s release.


(Errordeveloper) #8

Not much that I can see, but please take [quote=“justincormack, post:7, topic:8109”]
I cannot immediately see which change might have fixed this.
[/quote]

Some issue in Docker repository suggest that errors like this could be related to the storage driver, has that changed?


(Justin Cormack) #9

No there are no storage driver changes since the beta.


(Errordeveloper) #10

Will master make it to the release channel any time soon? I’d be happy to play around with snapshot builds :wink:


(Justin Cormack) #11

There will be a new beta on Tuesday.


(Frenchben) #12

@errordeveloper Can you confirm that the latest release solves your issue?


(Errordeveloper) #13

@frenchben @justincormack yes, I confirm that I’m not able to reproduce this issue with latest version.