Occasionally the Docker VM crashes and restarts when I run a container with a host mount.
Expected behavior
The container starts and the volume is mounted from the host.
Actual behavior
Docker fails with:
docker: Error response from daemon: stat /Users/me/my/host/volume: transport endpoint is not connected.
Logs
Docker[560] <Notice>: Creating resource Entry(3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b)
Docker[560] <Notice>: Write offset=0 data=[3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b:/Users/acotten/git/ansible-server/playbooks/] to file
Docker[560] <Notice>: Volume.start 3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b (paths = [/Users/acotten/git/ansible-server/playbooks/])
Docker[560] <Notice>: Created instance 3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b
Docker[560] <Error>: Fatal unexpected exception: Invalid_argument("String.sub / Bytes.sub")
Docker[557] <Notice>: Reap com.docker.osxfs (pid 560): exit status 1
Docker[561] <Notice>: Creating resource Entry(tcp:0.0.0.0:22)
Docker[561] <Notice>: Write offset=0 data=[tcp:0.0.0.0:22:00010016] to file
Docker[7371] <Notice>: Client reports version 1, commit 0123456789012345678901234567890123456789
Docker[561] <Notice>: Client.negotiate: received ((magic VMN3T)(version 15)(commit 4151b0fc71e51fb8e551d0a1870893ac99a379bc))
Docker[7371] <Critical>: sendmsg ok
Docker[561] <Notice>: Created instance tcp:0.0.0.0:22:00010016
Docker[557] <Notice>: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 563)
Docker[557] <Notice>: Signal terminated to com.docker.driver.amd64-linux (pid 563)
Docker[563] <Notice>: sending SIGTERM to com.docker.hyperkit pid 566
Docker[563] <Notice>: virtio-net-ipc: initialising uuid=25a2c89d-e239-4322-9532-9f2be1c99239,path=/Users/acotten/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0
Docker[563] <Notice>: Interface will have uuid 25a2c89d-e239-4322-9532-9f2be1c99239
Docker[563] <Notice>: Connection established with MAC=c0:ff:ee:c0:ff:ee and MTU 1500
Docker[563] <Notice>: virtio-9p: initialising path=/Users/acotten/Library/Containers/com.docker.docker/Data/s40,tag=db
Docker[563] <Notice>: virtio-9p: initialising path=/Users/acotten/Library/Containers/com.docker.docker/Data/s51,tag=port
Docker[563] <Notice>: linkname /Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Docker[563] <Notice>: COM1 connected to /dev/ttys001
Docker[563] <Notice>: COM1 linked to /Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Docker[561] <Notice>: PPP.listen: closing connection
Docker[562] <Notice>: VM has shutdown at 2016-05-20 14:47:03 +0200 CEST
Docker[557] <Notice>: Reap com.docker.driver.amd64-linux (pid 563): exit status 0
Docker[541] <Notice>: dockerState = starting
Docker[557] <Notice>: Stop 2 children with order 2: com.docker.osxfs (pid 560), com.docker.slirp (pid 561)
Docker[557] <Notice>: Signal terminated to com.docker.slirp (pid 561)
Docker[557] <Notice>: Reap com.docker.slirp (pid 561): signal: terminated
Information
$ pinata diagnose -u
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160520-145505.tar.gz
Most specific failure is: No error was detected
Your unique id is: 04523388-9760-49F7-BE77-4BACFB44DA8F
Please quote this in all correspondence.
Steps to reproduce the behavior
docker run --rm -h crashtest -v $(pwd)/vol2:/vol2 -v $(pwd)/vol1:/vol1 alpine id
Haven’t found a way to reproduce consistently yet though.
@dancryer Could you please post some more information about what you are running when you see this issue? A Dockerfile, shell script, Makefile, or similar or a version control repository exhibiting the issues would be extremely helpful in helping to track this down.
Reran the scenario and diagnosed:
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160526-131733.tar.gz
Most specific failure is: No error was detected
Your unique id is: FAF129FB-F8F1-4957-BE25-726F287F9241
Please quote this in all correspondence.
I’m running against a Dockerfile (workdir /opt/app), linked with postgres:
@dsheets I’m also running into exactly the same issue too. It seems to be happening when there’s a lot of filesystem access. I’m running a PHP site that has an admin panel that does a lot of AJAX requests whilst starting up. I can’t actually run the admin without it failing, and crashing the VM.
$ pinata diagnose
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160526-111934.tar.gz
Most specific failure is: No error was detected
Would you like to upload log files? [Y/n]: Y
Your unique id is: DA9CF6F1-6501-42CE-8ECC-9A204F564BBB
Please quote this in all correspondence.
Relevant section of the logs seems to say the same thing as mentioned above:
26/05/2016 11:19:21.564 Docker[25734]: Unknown exception caught: End_of_file
26/05/2016 11:19:21.564 Docker[25734]: Fatal unexpected exception: End_of_file
26/05/2016 11:19:21.569 Docker[25611]: Reap com.docker.osxfs (pid 25734): exit status 1
26/05/2016 11:19:22.571 Docker[25611]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 25736)
26/05/2016 11:19:22.571 Docker[25611]: Signal terminated to com.docker.driver.amd64-linux (pid 25736)
26/05/2016 11:19:22.571 Docker[25736]: sending SIGTERM to com.docker.hyperkit pid 25738
Same here since v1.11.1-beta13. The project is at https://github.com/rslnd/rosalind - reliably crashes when triggering a reload via Meteor’s file watcher.
docker-compose
Exception in thread Thread-10: Traceback (most recent call last): File "threading.py", line 810, in __bootstrap_inner File "threading.py", line 763, in run File "compose/cli/log_printer.py", line 149, in tail_container_logs File "compose/cli/log_printer.py", line 179, in wait_on_exit File "compose/container.py", line 239, in wait File "site-packages/docker/utils/decorators.py", line 21, in wrapped File "site-packages/docker/api/container.py", line 439, in wait File "site-packages/docker/client.py", line 154, in _raise_for_status APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")
Console.app
26/05/16 1:18:41,312 PM Docker[36761]: Unknown exception caught: End_of_file 26/05/16 1:18:41,313 PM Docker[36761]: Fatal unexpected exception: End_of_file 26/05/16 1:18:41,321 PM Docker[30914]: Reap com.docker.osxfs (pid 36761): exit status 1 26/05/16 1:18:42,323 PM Docker[30914]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 36765) 26/05/16 1:18:42,324 PM Docker[30914]: Signal terminated to com.docker.driver.amd64-linux (pid 36765) 26/05/16 1:18:42,325 PM Docker[36765]: sending SIGTERM to com.docker.hyperkit pid 36769
Pinata says:
OS X: version 10.11.5 (build: 15F34) Docker.app: version v1.11.1-beta13 Running diagnostic tests: [OK] Moby booted [OK] driver.amd64-linux [OK] vmnetd [OK] osxfs [OK] db [OK] slirp [OK] menubar [OK] environment [OK] Docker [OK] VT-x Docker logs are being collected into /tmp/20160526-132500.tar.gz Most specific failure is: No error was detected Your unique id is: 04383007-FB14-4141-A785-EE24AA45F9E5 Please quote this in all correspondence.
@dsheets I’m running a custom PHP-based CMS (https://github.com/block8/octo) in a container with latest stable nginx and PHP7 FPM. The container is based on Ubuntu 16.04.
Like others, it appears that the error occurs under heavier IO workload, I think primarily reads. I’ll try and make it happen in a simpler container and share the Dockerfile here.
It runs PHP under Nginx, the script reads a volume-mounted directory and writes the name of each file it encounters to a file in the mounted volume. With enough requests and enough concurrency, it crashes every time.
Edit: I ran Pinata after my latest crash…
`OS X: version 10.11.3 (build: 15D21)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160526-161944.tar.gz
Most specific failure is: No error was detected
Would you like to upload log files? [Y/n]: Y
Your unique id is: 3D171D36-D40D-4841-9513-95E83A2D09C4
Please quote this in all correspondence.`
rm: cannot remove 'shared/css/cyph.modest.css.map': Software caused connection abort
rm: cannot remove 'shared/css/fonts.css.map': Transport endpoint is not connected
rm: cannot remove 'shared/css/loading.css.map': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/analytics.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/base.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/crypto/crypto.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/session/session.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph.com/main.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph.im/main.js': Transport endpoint is not connected
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160527-113446.tar.gz
Most specific failure is: No error was detected
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13.1
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160530-130905.tar.gz
Most specific failure is: No error was detected
Your unique id is: FAF129FB-F8F1-4957-BE25-726F287F9241
Please quote this in all correspondence.
I am still running into this issue.
I have directories mounted into the containers.
I am using facebook-watchman to trigger make builds and it crashes docker.
Docker for Mac: version: mac-v1.12.0-beta16.2
OS X: version 10.11.4 (build: 15E65)
logs: /tmp/20160627-110059.tar.gz
failure: No error was detected
[OK] docker-cli
[OK] app
[OK] menubar
[OK] virtualization
[OK] system
[OK] osxfs
[OK] db
[OK] slirp
[OK] moby-console
[OK] logs
[OK] vmnetd
[OK] env
[OK] moby
[OK] driver.amd64-linux
Makefile:833: .deps/xlua.Plo: Software caused connection abort
Makefile:834: .deps/xmalloc.Plo: Transport endpoint is not connected
Makefile:835: .deps/xsignal.Plo: Transport endpoint is not connected
Makefile:836: .deps/xstring.Plo: Transport endpoint is not connected
Makefile:837: .deps/xtree.Plo: Transport endpoint is not connected
Version:
Docker for Mac: version: mac-v1.12.0-beta20
OS X: version 10.11.5 (build: 15F34)
Ok in v1.12.0-beta21 in a heavy build, I’m seeing Transport endpoint not connected. We are building boost in a container
rsync -a --delete /Users/rich/ws/git/src/externals/boost/boost/ /Users/rich/ws/var/debug/local/built/src/externals/boost/boost/
rsync: send_files failed to open "/Users/rich/ws/git/src/externals/boost/boost/libs/preprocessor/include/boost/preprocessor/iteration/detail/bounds/lower4.hpp": Transport endpoint is not connected (107)
rsync: send_files failed to open "/Users/rich/ws/git/src/externals/boost/boost/libs/preprocessor/include/boost/preprocessor/iteration/detail/bounds/lower5.hpp": Transport endpoint is not connected (107)
It would tremendously aid diagnosis and resolution of this issue if you (and other recent reporters of this symptom) would post your diagnostic ID from the Diagnose & Feedback window of the application. Unfortunately, you’ll also need to at least click on the button that says Open Issues in order to upload your diagnostic bundle.
Additionally, any reproduction cases would be extremely helpful as we can then more easily instrument the application to debug the root cause.
There are potentially many different causes of this kind of failure and examining the logs and then instrumentation with a reliable reproducer is the best way to make progress toward a fix.
Thanks for your report and for participating in the Docker for Mac Beta!
Super happy to help. This looks like it happens in a build of boost. So I can cut that out of our build step and see if we can give u a reproducible case.
I didn’t know about the diagnose and feedback window but will run again and provide that.
Understand how hard these are to debug. The osxfs is under heavy pressure with a boost build I know.
Thanks,
Rich
No! Try not. Do, or do not. There is no try. Master Yoda. Star Wars Episode V: The Empire Strikes Back
