Docker Community Forums

Share and learn in the Docker community.

Crash when running container with host mount

beta

(Antoine Cotten) #1

Occasionally the Docker VM crashes and restarts when I run a container with a host mount.

Expected behavior

The container starts and the volume is mounted from the host.

Actual behavior

Docker fails with:

docker: Error response from daemon: stat /Users/me/my/host/volume: transport endpoint is not connected.

Logs

Docker[560] <Notice>: Creating resource Entry(3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b)
Docker[560] <Notice>: Write offset=0 data=[3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b:/Users/acotten/git/ansible-server/playbooks/] to file
Docker[560] <Notice>: Volume.start 3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b (paths = [/Users/acotten/git/ansible-server/playbooks/])
Docker[560] <Notice>: Created instance 3f9f5c930c7c856a36843c92d4e9e229ca3dbd8584ad165863e25212e14df73b
Docker[560] <Error>: Fatal unexpected exception: Invalid_argument("String.sub / Bytes.sub")
Docker[557] <Notice>: Reap com.docker.osxfs (pid 560): exit status 1
Docker[561] <Notice>: Creating resource Entry(tcp:0.0.0.0:22)
Docker[561] <Notice>: Write offset=0 data=[tcp:0.0.0.0:22:00010016] to file
Docker[7371] <Notice>: Client reports version 1, commit 0123456789012345678901234567890123456789
Docker[561] <Notice>: Client.negotiate: received ((magic VMN3T)(version 15)(commit 4151b0fc71e51fb8e551d0a1870893ac99a379bc))
Docker[7371] <Critical>: sendmsg ok
Docker[561] <Notice>: Created instance tcp:0.0.0.0:22:00010016
Docker[557] <Notice>: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 563)
Docker[557] <Notice>: Signal terminated to com.docker.driver.amd64-linux (pid 563)
Docker[563] <Notice>: sending SIGTERM to com.docker.hyperkit pid 566 
Docker[563] <Notice>: virtio-net-ipc: initialising uuid=25a2c89d-e239-4322-9532-9f2be1c99239,path=/Users/acotten/Library/Containers/com.docker.docker/Data/s50,macfile=/Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/mac.0
Docker[563] <Notice>: Interface will have uuid 25a2c89d-e239-4322-9532-9f2be1c99239
Docker[563] <Notice>: Connection established with MAC=c0:ff:ee:c0:ff:ee and MTU 1500
Docker[563] <Notice>: virtio-9p: initialising path=/Users/acotten/Library/Containers/com.docker.docker/Data/s40,tag=db
Docker[563] <Notice>: virtio-9p: initialising path=/Users/acotten/Library/Containers/com.docker.docker/Data/s51,tag=port
Docker[563] <Notice>: linkname /Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Docker[563] <Notice>: COM1 connected to /dev/ttys001
Docker[563] <Notice>: COM1 linked to /Users/acotten/Library/Containers/com.docker.docker/Data/com.docker.driver.amd64-linux/tty
Docker[561] <Notice>: PPP.listen: closing connection
Docker[562] <Notice>: VM has shutdown at 2016-05-20 14:47:03 +0200 CEST
Docker[557] <Notice>: Reap com.docker.driver.amd64-linux (pid 563): exit status 0
Docker[541] <Notice>: dockerState = starting
Docker[557] <Notice>: Stop 2 children with order 2: com.docker.osxfs (pid 560), com.docker.slirp (pid 561)
Docker[557] <Notice>: Signal terminated to com.docker.slirp (pid 561)
Docker[557] <Notice>: Reap com.docker.slirp (pid 561): signal: terminated

Information

$ pinata diagnose -u

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160520-145505.tar.gz
Most specific failure is: No error was detected
Your unique id is: 04523388-9760-49F7-BE77-4BACFB44DA8F
Please quote this in all correspondence.

Steps to reproduce the behavior

docker run --rm -h crashtest -v $(pwd)/vol2:/vol2 -v $(pwd)/vol1:/vol1 alpine id

Haven’t found a way to reproduce consistently yet though.


(Mixman) #2

Same issue. Starting, then stopping, then starting the same container fails killing the vm and thus all running containers.

May 25 00:14:25 FL701 Docker[36474] <Notice>: Docker is responding
May 25 00:14:25 FL701 Docker[10802] <Notice>: VM has started at 2016-05-25 00:14:25 +0300 EEST
May 25 00:14:25 FL701 Docker[10763] <Notice>: dockerState = running
May 25 00:14:43 FL701 Docker[36472] <Notice>: Creating resource Entry(90d26a4dee52680b0a277027c11e922f462f76369ccf4313fa5cea50497fed7e)
May 25 00:14:43 FL701 Docker[36472] <Notice>: Write offset=0 data=[90d26a4dee52680b0a277027c11e922f462f76369ccf4313fa5cea50497fed7e:/Users/mixman/code/project/beta/:/Users/mixman/code/project-ui/] to file
May 25 00:14:43 FL701 Docker[36472] <Notice>: Volume.start 90d26a4dee52680b0a277027c11e922f462f76369ccf4313fa5cea50497fed7e (paths = [/Users/mixman/code/project/beta/, /Users/mixman/code/project-ui/])
May 25 00:14:43 FL701 Docker[36472] <Notice>: Created instance 90d26a4dee52680b0a277027c11e922f462f76369ccf4313fa5cea50497fed7e
May 25 00:14:43 FL701 Docker[36473] <Notice>: Creating resource Entry(tcp:0.0.0.0:8000)
May 25 00:14:43 FL701 Docker[36473] <Notice>: Write offset=0 data=[tcp:0.0.0.0:8000:00011f40] to file
May 25 00:14:43 FL701 Docker[36473] <Notice>: Created instance tcp:0.0.0.0:8000:00011f40
May 25 00:27:36 FL701 Docker[36472] <Notice>: Volume.stop 90d26a4dee52680b0a277027c11e922f462f76369ccf4313fa5cea50497fed7e (paths = [/Users/mixman/code/project/beta/, /Users/mixman/code/project-ui/])
May 25 00:27:36 FL701 Docker[36472] <Error>: WATCH SUBSCRIBER REMOVAL INVARIANT VIOLATED
--- last message repeated 1 time ---
May 25 00:27:38 FL701 Docker[36472] <Notice>: Creating resource Entry(3023cae28828b73b7b8bb2b3c290471067626ea512ed171a03d7cd274a906a22)
May 25 00:27:38 FL701 Docker[36472] <Notice>: Write offset=0 data=[3023cae28828b73b7b8bb2b3c290471067626ea512ed171a03d7cd274a906a22:/Users/mixman/code/project/beta/:/Users/mixman/code/project-ui/] to file
May 25 00:27:38 FL701 Docker[36472] <Notice>: Volume.start 3023cae28828b73b7b8bb2b3c290471067626ea512ed171a03d7cd274a906a22 (paths = [/Users/mixman/code/project/beta/, /Users/mixman/code/project-ui/])
May 25 00:27:38 FL701 Docker[36472] <Notice>: Created instance 3023cae28828b73b7b8bb2b3c290471067626ea512ed171a03d7cd274a906a22
May 25 00:27:38 FL701 Docker[36472] <Error>: Fatal unexpected exception: Invalid_argument("String.sub / Bytes.sub")
May 25 00:27:38 FL701 Docker[10796] <Notice>: Reap com.docker.osxfs (pid 36472): exit status 1
May 25 00:27:38 FL701 Docker[36473] <Notice>: Creating resource Entry(tcp:0.0.0.0:8000)
May 25 00:27:38 FL701 Docker[36473] <Notice>: Write offset=0 data=[tcp:0.0.0.0:8000:00011f40] to file
May 25 00:27:38 FL701 Docker[36473] <Notice>: Created instance tcp:0.0.0.0:8000:00011f40
May 25 00:27:39 FL701 Docker[10796] <Notice>: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 36474)
May 25 00:27:39 FL701 Docker[10796] <Notice>: Signal terminated to com.docker.driver.amd64-linux (pid 36474)
May 25 00:27:39 FL701 Docker[36474] <Notice>: sending SIGTERM to com.docker.hyperkit pid 36476

Versions:

OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta12
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x

(Dancryer) #3

I’m also getting this issue, in beta 13 for Mac.

The most relevant log lines appear to be these:

26/05/2016 09:28:58.517 Docker[58145]: Unknown exception caught: End_of_file 26/05/2016 09:28:58.517 Docker[58145]: Fatal unexpected exception: End_of_file 26/05/2016 09:28:58.520 Docker[57991]: Reap com.docker.osxfs (pid 58145): exit status 1 26/05/2016 09:28:59.525 Docker[57991]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 58147) 26/05/2016 09:28:59.525 Docker[57991]: Signal terminated to com.docker.driver.amd64-linux (pid 58147) 26/05/2016 09:28:59.525 Docker[58147]: sending SIGTERM to com.docker.hyperkit pid 58149

I’ve tried rebuilding the image, restarting the container, etc. but to no avail. :frowning:


(David Sheets) #4

@dancryer Could you please post some more information about what you are running when you see this issue? A Dockerfile, shell script, Makefile, or similar or a version control repository exhibiting the issues would be extremely helpful in helping to track this down.


(David Sheets) #5

@mixman Could you please post your unique error report identifier that’s printed when you run pinata diagnose -u?


(David Sheets) #6

Hi Antoine,

Thanks for the issue report. I’m reviewing your error logs now.

David


(Mixman) #7

Reran the scenario and diagnosed:
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160526-131733.tar.gz
Most specific failure is: No error was detected
Your unique id is: FAF129FB-F8F1-4957-BE25-726F287F9241
Please quote this in all correspondence.

I’m running against a Dockerfile (workdir /opt/app), linked with postgres:

docker run --rm -it -p 8000:8000 --name project
-e DB_USER=postgres
-v ~/code/project/subdir/:/opt/app/:rw
-v ~/code/project-ui/:/opt/project-ui/:rw
–link postgres:postgres
company/project


(Elliot Wright) #8

@dsheets I’m also running into exactly the same issue too. It seems to be happening when there’s a lot of filesystem access. I’m running a PHP site that has an admin panel that does a lot of AJAX requests whilst starting up. I can’t actually run the admin without it failing, and crashing the VM.

$ pinata diagnose
OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160526-111934.tar.gz
Most specific failure is: No error was detected
Would you like to upload log files? [Y/n]: Y

Your unique id is: DA9CF6F1-6501-42CE-8ECC-9A204F564BBB
Please quote this in all correspondence.

Relevant section of the logs seems to say the same thing as mentioned above:

26/05/2016 11:19:21.564 Docker[25734]: Unknown exception caught: End_of_file
26/05/2016 11:19:21.564 Docker[25734]: Fatal unexpected exception: End_of_file
26/05/2016 11:19:21.569 Docker[25611]: Reap com.docker.osxfs (pid 25734): exit status 1
26/05/2016 11:19:22.571 Docker[25611]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 25736)
26/05/2016 11:19:22.571 Docker[25611]: Signal terminated to com.docker.driver.amd64-linux (pid 25736)
26/05/2016 11:19:22.571 Docker[25736]: sending SIGTERM to com.docker.hyperkit pid 25738

If you need any more info, just let me know! :slight_smile:


(Albertzak) #9

Same here since v1.11.1-beta13. The project is at https://github.com/rslnd/rosalind - reliably crashes when triggering a reload via Meteor’s file watcher.

docker-compose

Exception in thread Thread-10: Traceback (most recent call last): File "threading.py", line 810, in __bootstrap_inner File "threading.py", line 763, in run File "compose/cli/log_printer.py", line 149, in tail_container_logs File "compose/cli/log_printer.py", line 179, in wait_on_exit File "compose/container.py", line 239, in wait File "site-packages/docker/utils/decorators.py", line 21, in wrapped File "site-packages/docker/api/container.py", line 439, in wait File "site-packages/docker/client.py", line 154, in _raise_for_status APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")

Console.app

26/05/16 1:18:41,312 PM Docker[36761]: Unknown exception caught: End_of_file 26/05/16 1:18:41,313 PM Docker[36761]: Fatal unexpected exception: End_of_file 26/05/16 1:18:41,321 PM Docker[30914]: Reap com.docker.osxfs (pid 36761): exit status 1 26/05/16 1:18:42,323 PM Docker[30914]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 36765) 26/05/16 1:18:42,324 PM Docker[30914]: Signal terminated to com.docker.driver.amd64-linux (pid 36765) 26/05/16 1:18:42,325 PM Docker[36765]: sending SIGTERM to com.docker.hyperkit pid 36769

Pinata says:

OS X: version 10.11.5 (build: 15F34) Docker.app: version v1.11.1-beta13 Running diagnostic tests: [OK] Moby booted [OK] driver.amd64-linux [OK] vmnetd [OK] osxfs [OK] db [OK] slirp [OK] menubar [OK] environment [OK] Docker [OK] VT-x Docker logs are being collected into /tmp/20160526-132500.tar.gz Most specific failure is: No error was detected Your unique id is: 04383007-FB14-4141-A785-EE24AA45F9E5 Please quote this in all correspondence.

Thanks a lot! :+1:


(Dancryer) #10

@dsheets I’m running a custom PHP-based CMS (https://github.com/block8/octo) in a container with latest stable nginx and PHP7 FPM. The container is based on Ubuntu 16.04.

Like others, it appears that the error occurs under heavier IO workload, I think primarily reads. I’ll try and make it happen in a simpler container and share the Dockerfile here.


(Dancryer) #11

@dsheets This isn’t as clean of a reproducible test case as I would have liked, but this causes the crash with 100% failure rate for me: https://github.com/dancryer/break-docker-for-mac

Instructions to run are in the README on Github.

It runs PHP under Nginx, the script reads a volume-mounted directory and writes the name of each file it encounters to a file in the mounted volume. With enough requests and enough concurrency, it crashes every time.

Edit: I ran Pinata after my latest crash…

`OS X: version 10.11.3 (build: 15D21)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160526-161944.tar.gz
Most specific failure is: No error was detected
Would you like to upload log files? [Y/n]: Y

Your unique id is: 3D171D36-D40D-4841-9513-95E83A2D09C4
Please quote this in all correspondence.`


(David Ham) #12

This sounds a lot like my issue, posted about an hour ago:


(Cyph, Inc.) #13

+1; fairly consistently reproducible here.

rm: cannot remove 'shared/css/cyph.modest.css.map': Software caused connection abort
rm: cannot remove 'shared/css/fonts.css.map': Transport endpoint is not connected
rm: cannot remove 'shared/css/loading.css.map': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/analytics.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/base.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/crypto/crypto.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph/session/session.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph.com/main.js': Transport endpoint is not connected
rm: cannot remove 'shared/js/cyph.im/main.js': Transport endpoint is not connected
OS X: version 10.11.4 (build: 15E65)
Docker.app: version v1.11.1-beta13
Running diagnostic tests:
[OK]      Moby booted
[OK]      driver.amd64-linux
[OK]      vmnetd
[OK]      osxfs
[OK]      db
[OK]      slirp
[OK]      menubar
[OK]      environment
[OK]      Docker
[OK]      VT-x
Docker logs are being collected into /tmp/20160527-113446.tar.gz
Most specific failure is: No error was detected

(Mixman) #14

Still an issue with 15F34.

OS X: version 10.11.5 (build: 15F34)
Docker.app: version v1.11.1-beta13.1
Running diagnostic tests:
[OK] Moby booted
[OK] driver.amd64-linux
[OK] vmnetd
[OK] osxfs
[OK] db
[OK] slirp
[OK] menubar
[OK] environment
[OK] Docker
[OK] VT-x
Docker logs are being collected into /tmp/20160530-130905.tar.gz
Most specific failure is: No error was detected
Your unique id is: FAF129FB-F8F1-4957-BE25-726F287F9241
Please quote this in all correspondence.


(James Randall Qualls) #16

I am still running into this issue.
I have directories mounted into the containers.
I am using facebook-watchman to trigger make builds and it crashes docker.

Docker for Mac: version: mac-v1.12.0-beta16.2
OS X: version 10.11.4 (build: 15E65)
logs: /tmp/20160627-110059.tar.gz
failure: No error was detected
[OK] docker-cli
[OK] app
[OK] menubar
[OK] virtualization
[OK] system
[OK] osxfs
[OK] db
[OK] slirp
[OK] moby-console
[OK] logs
[OK] vmnetd
[OK] env
[OK] moby
[OK] driver.amd64-linux


(Josh Reichardt) #17

Just experienced this as well when attempting to watch a directory for changes on the host with webpack.


(Gaijin03) #18

Looks like I’m seeing this as well. Seems to happen when there is a heavy load (make -j).

I’m running setup.pl from https://github.com/gaijin03/fedtest

Makefile:833: .deps/xlua.Plo: Software caused connection abort
Makefile:834: .deps/xmalloc.Plo: Transport endpoint is not connected
Makefile:835: .deps/xsignal.Plo: Transport endpoint is not connected
Makefile:836: .deps/xstring.Plo: Transport endpoint is not connected
Makefile:837: .deps/xtree.Plo: Transport endpoint is not connected

Version:

Docker for Mac: version: mac-v1.12.0-beta20
OS X: version 10.11.5 (build: 15F34)

(Richard Tong) #19

Ok in v1.12.0-beta21 in a heavy build, I’m seeing Transport endpoint not connected. We are building boost in a container

rsync -a --delete /Users/rich/ws/git/src/externals/boost/boost/ /Users/rich/ws/var/debug/local/built/src/externals/boost/boost/
rsync: send_files failed to open "/Users/rich/ws/git/src/externals/boost/boost/libs/preprocessor/include/boost/preprocessor/iteration/detail/bounds/lower4.hpp": Transport endpoint is not connected (107)
rsync: send_files failed to open "/Users/rich/ws/git/src/externals/boost/boost/libs/preprocessor/include/boost/preprocessor/iteration/detail/bounds/lower5.hpp": Transport endpoint is not connected (107)

(David Sheets) #20

Hi Richard,

It would tremendously aid diagnosis and resolution of this issue if you (and other recent reporters of this symptom) would post your diagnostic ID from the Diagnose & Feedback window of the application. Unfortunately, you’ll also need to at least click on the button that says Open Issues in order to upload your diagnostic bundle.

Additionally, any reproduction cases would be extremely helpful as we can then more easily instrument the application to debug the root cause.

There are potentially many different causes of this kind of failure and examining the logs and then instrumentation with a reliable reproducer is the best way to make progress toward a fix.

Thanks for your report and for participating in the Docker for Mac Beta!

David


(Richard Tong) #21

Super happy to help. This looks like it happens in a build of boost. So I can cut that out of our build step and see if we can give u a reproducible case.

I didn’t know about the diagnose and feedback window but will run again and provide that.

Understand how hard these are to debug. The osxfs is under heavy pressure with a boost build I know.

Thanks,
Rich

No! Try not. Do, or do not. There is no try. Master Yoda. Star Wars Episode V: The Empire Strikes Back

http://tongfamily.com | @richtong | http://fb.me/richtong1 | +1-206-963-9029