Docker for Mac 1.11.2-beta15 crashes with many I/O-Operations

Expected behavior

Docker should not crash with many I/O-Operations

Actual behavior

Docker crashes with many I/O-Operations

Information

Docker for Mac: version: mac-v1.11.2-beta15
OS X: version 10.11.5 (build: 15F34)
logs: /tmp/20160614-120913.tar.gz
failure: docker ps failed: Failure(“docker ps: timeout after 10.00s”)
[ERROR] docker-cli
docker ps failed
[OK] app
[OK] menubar
[OK] virtualization
[OK] system
[OK] osxfs
[OK] db
[OK] slirp
[OK] moby-console
[OK] logs
[OK] vmnetd
[OK] env
[OK] moby
[OK] driver.amd64-linux

Diagnostic ID:
0AA6A147-7B5D-4FC5-B60F-41EEDDB30DAC

Steps to reproduce the behavior

  1. Start a container
  2. Generate many I/O-Operations (e.g import a large DB-Dump into a mysql-container)

My docker also crashes after intesive writes to my mysql database.
I think this is the relevant part of the log:

14.06.16 12:27:27,406 Docker[372]: name:NSInternalInconsistencyException
14.06.16 12:27:27,406 Docker[372]: reason:Failed to set posix_spawn_file_actions for fd -1 at index 0 with errno 9
14.06.16 12:27:27,408 Docker[372]: return adress:[140735638021338, 140735440576382, 140735638443197, 140735474149469, 4369441143, 4369455763, 140735415482685, 140735415436299, 140735415452315, 140735415450368, 140735614084318, 140735614075713]
14.06.16 12:27:27,408 Docker[372]: function:0   CoreFoundation                      0x00007fff91b624f2 __exceptionPreprocess + 178
14.06.16 12:27:27,409 Docker[372]: function:1   libobjc.A.dylib                     0x00007fff85f15f7e objc_exception_throw + 48
14.06.16 12:27:27,409 Docker[372]: function:2   CoreFoundation                      0x00007fff91bc94bd +[NSException raise:format:] + 205
14.06.16 12:27:27,409 Docker[372]: function:3   Foundation                          0x00007fff87f1a85d -[NSConcreteTask launchWithDictionary:] + 2494
14.06.16 12:27:27,409 Docker[372]: function:4   Docker                              0x0000000104706177 _TTSf4gs_d___TZFC6Docker7Backend15sendRequestSyncfSST5errorGSqSS_4respSS_ + 791
14.06.16 12:27:27,409 Docker[372]: function:5   Docker                              0x0000000104709a93 _TFZFC6Docker7Backend27listenForDockerStateChangesFT_T_U_FT_T_ + 1459
14.06.16 12:27:27,409 Docker[372]: function:6   libdispatch.dylib                   0x00007fff8472793d _dispatch_call_block_and_release + 12
14.06.16 12:27:27,409 Docker[372]: function:7   libdispatch.dylib                   0x00007fff8471c40b _dispatch_client_callout + 8
14.06.16 12:27:27,409 Docker[372]: *** Terminating app due to uncaught exception 'NSInternalInconsistencyException', reason: 'Failed to set posix_spawn_file_actions for fd -1 at index 0 with errno 9'
*** First throw call stack:
(
    0   CoreFoundation                      0x00007fff91b624f2 __exceptionPreprocess + 178
    1   libobjc.A.dylib                     0x00007fff85f15f7e objc_exception_throw + 48
    2   CoreFoundation                      0x00007fff91bc94bd +[NSException raise:format:] + 205
    3   Foundation                          0x00007fff87f1a85d -[NSConcreteTask launchWithDictionary:] + 2494
    4   Docker                              0x0000000104706177 _TTSf4gs_d___TZFC6Docker7Backend15sendRequestSyncfSST5errorGSqSS_4respSS_ + 791
    5   Docker                              0x0000000104709a93 _TFZFC6Docker7Backend27listenForDockerStateChangesFT_T_U_FT_T_ + 1459
    6   libdispatch.dylib                   0x00007fff8472793d _dispatch_call_block_and_release + 12
    7   libdispatch.dylib                   0x00007fff8471c40b _dispatch_client_callout + 8
    8   libdispatch.dylib                   0x00007fff8472029b _dispatch_root_queue_drain + 1890
    9   libdispatch.dylib                   0x00007fff8471fb00 _dispatch_worker_thread3 + 91
    10  libsystem_pthread.dylib             0x00007fff9048e4de _pthread_wqthread + 1129
    11  libsystem_pthread.dylib             0x00007fff9048c341 start_wqthread + 13
)
14.06.16 12:27:27,410 Docker[372]: function:8   libdispatch.dylib                   0x00007fff8472029b _dispatch_root_queue_drain + 1890
14.06.16 12:27:27,410 Docker[372]: function:9   libdispatch.dylib                   0x00007fff8471fb00 _dispatch_worker_thread3 + 91
14.06.16 12:27:27,410 Docker[372]: function:10  libsystem_pthread.dylib             0x00007fff9048e4de _pthread_wqthread + 1129
14.06.16 12:27:27,410 Docker[372]: function:11  libsystem_pthread.dylib             0x00007fff9048c341 start_wqthread + 13
14.06.16 12:27:27,410 Docker[372]: Signal SIGABRT caught, terminating
14.06.16 12:27:27,410 Docker[372]: Terminating com.docker.osx.hyperkit.linux and waiting until exit
14.06.16 12:27:27,414 Docker[398]: Received SIGTERM, shutting down
14.06.16 12:27:27,415 Docker[398]: Stop 1 children with order 1: com.docker.driver.amd64-linux (pid 408)
14.06.16 12:27:27,415 Docker[398]: Signal terminated to com.docker.driver.amd64-linux (pid 408)
14.06.16 12:27:27,417 Docker[408]: sending SIGTERM to com.docker.hyperkit pid 412
14.06.16 12:27:27,432 Docker[398]: EOF on watchdog pipe: shutting down`

Since Beta 15, I’m getting “too many open files” from the host osx after heavy DB I/O (mysql, oracle), soon after that Docker for Mac crashes. Could be related.

OSX: 10.11.5

Wasn’t allowed to attach files to the post… please find a truncated version of the crash report:

Process: Docker [41953]
Path: /Applications/Docker.app/Contents/MacOS/Docker
Identifier: com.docker.docker
Version: 1.11.2-beta15 (9168)
Code Type: X86-64 (Native)
Parent Process: ??? [1]
Responsible: Docker [41953]
User ID: 501

Date/Time: 2016-06-14 15:18:56.216 +0200
OS Version: Mac OS X 10.11.5 (15F34)
Report Version: 11
Anonymous UUID: 9B6E2B0D-059C-DC7B-A93E-222515899820

Sleep/Wake UUID: 9B6F24CF-2F1F-4073-8CFA-03FE7B6AECB7

Time Awake Since Boot: 190000 seconds
Time Since Wake: 10000 seconds

System Integrity Protection: enabled

Crashed Thread: 2 Dispatch queue: com.apple.root.background-qos

Exception Type: EXC_CRASH (SIGABRT)
Exception Codes: 0x0000000000000000, 0x0000000000000000
Exception Note: EXC_CORPSE_NOTIFY

Application Specific Information:
*** Terminating app due to uncaught exception ‘NSInternalInconsistencyException’, reason: 'Failed to set posix_spawn_file_actions for fd -1 at index 2 with errno 9’
terminating with uncaught exception of type NSException
abort() called

Application Specific Backtrace 1:
0 CoreFoundation 0x00007fff989484f2 __exceptionPreprocess + 178
1 libobjc.A.dylib 0x00007fff95964f7e objc_exception_throw + 48
2 CoreFoundation 0x00007fff989af4bd +[NSException raise:format:] + 205
3 Foundation 0x00007fff943d085d -[NSConcreteTask launchWithDictionary:] + 2494
4 Docker 0x00000001043ee177 TTSf4gs_d___TZFC6Docker7Backend15sendRequestSyncfSST5errorGSqSS_4respSS + 791
5 Docker 0x00000001043f1a93 TFZFC6Docker7Backend27listenForDockerStateChangesFT_T_U_FT_T + 1459
6 libdispatch.dylib 0x00007fff94ebb93d _dispatch_call_block_and_release + 12
7 libdispatch.dylib 0x00007fff94eb040b _dispatch_client_callout + 8
8 libdispatch.dylib 0x00007fff94eb429b _dispatch_root_queue_drain + 1890
9 libdispatch.dylib 0x00007fff94eb3b00 _dispatch_worker_thread3 + 91
10 libsystem_pthread.dylib 0x00007fff9b98b4de _pthread_wqthread + 1129
11 libsystem_pthread.dylib 0x00007fff9b989341 start_wqthread + 13

Global Trace Buffer (reverse chronological seconds):
0.889647 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532f94cd0) failed
0.889938 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb53533a210) failed
0.890563 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352b01c0) failed
0.890695 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352b0770) failed
0.890826 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8cb40) failed
0.890938 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352af8b0) failed
0.891074 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352b1290) failed
0.891204 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b160) failed
0.891321 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b2e0) failed
0.891414 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b2e0) failed
0.891529 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8bd60) failed
0.891661 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b160) failed
0.891779 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b150) failed
0.891901 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532e8b130) failed
0.892054 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352b0350) failed
0.892410 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb532f94cd0) failed
0.892589 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352afb40) failed
147.500089 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352b2b30) failed
153.934476 libsystem_trace.dylib 0x00007fff9b6b30fa dyld_image_header_containing_address(0x7fb5352aafa0) failed
1054.548247 CFNetwork 0x00007fff857413cb TCP Conn 0x7fb53526aed0 SSL Handshake DONE
1054.623439 CFNetwork 0x00007fff857412a7 TCP Conn 0x7fb53526aed0 starting SSL negotiation
1054.623620 CFNetwork 0x00007fff8573fc71 TCP Conn 0x7fb53526aed0 complete. fd: 1313, err: 0
1054.624065 CFNetwork 0x00007fff857ce54b TCP Conn 0x7fb53526aed0 event 1. err: 0
1054.686833 CFNetwork 0x00007fff8573ef43 TCP Conn 0x7fb53526aed0 started
2228.930569 CFNetwork 0x00007fff857413cb TCP Conn 0x7fb5352758e0 SSL Handshake DONE
2229.161055 CFNetwork 0x00007fff857412a7 TCP Conn 0x7fb5352758e0 starting SSL negotiation

External Modification Summary:
Calls made by other processes targeting this process:
task_for_pid: 1683
thread_create: 0
thread_set_state: 0
Calls made by this process:
task_for_pid: 0
thread_create: 0
thread_set_state: 0
Calls made by all processes on this machine:
task_for_pid: 11391567
thread_create: 0
thread_set_state: 0

VM Region Summary:
ReadOnly portion of Libraries: Total=294.3M resident=0K(0%) swapped_out_or_unallocated=294.3M(100%)
Writable regions: Total=93.7M written=0K(0%) resident=0K(0%) swapped_out=0K(0%) unallocated=93.7M(100%)

                            VIRTUAL   REGION 

REGION TYPE SIZE COUNT (non-coalesced)
=========== ======= =======
Accelerate.framework 256K 3
Activity Tracing 2048K 2
CG backing stores 1556K 5
CG image 256K 36
CG shared images 272K 9
CoreAnimation 192K 30
CoreUI image data 2036K 18
CoreUI image file 224K 5
Dispatch continuations 8192K 2
Foundation 24K 3
Image IO 100K 2
Kernel Alloc Once 8K 3
MALLOC 47.9M 36
MALLOC guard page 32K 7
Memory Tag 242 12K 2
OpenCL 24K 4
Process Corpse Info 2048K 2
STACK GUARD 56.0M 13
Stack 13.1M 13
VM_ALLOCATE 180K 26
__DATA 28.7M 282
__IMAGE 528K 2
__LINKEDIT 95.6M 20
__TEXT 198.7M 288
__UNICODE 552K 2
mapped file 143.0M 42
shared memory 16.3M 11
=========== ======= =======
TOTAL 617.4M 841

I’m seeing this behavior as well with the same error message as reported above. I can’t associate it with high I/O – docker beta seems to spontaneously terminate even when the containers seem idle.