Exception in thread Thread-N in Beta 13

fskreuz · May 26, 2016, 3:59pm

Expected behavior

Docker should be running fine.

Actual behavior

Docker containers die. Error shown in terminal.

Information

See log at bottom. Was running a php-apache, php-cli and a mysql container (not in particular order).

Terminal output:

Exception in thread Thread-5:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")

Exception in thread Thread-4:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")

Exception in thread Thread-6:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/me/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

danwahlin · June 13, 2016, 4:23am

I’m seeing this same behavior in Beta 15 as well: Docker for Mac: version: mac-v1.11.2-beta15
OS X: version 10.11.5 (build: 15F34). Here’s the specific error I see which is very similar (several others that are similar display in the console as well):

Exception in thread Thread-8:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")
Exception in thread Thread-10:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")

If there’s any additional I can provide to help identify the problem please let me know. Since Docker crashes entirely I can’t run diagnostics until I restart it. At that point the diagnostics say everything is “OK”.

v9chris · June 16, 2016, 8:10am

I’m getting the exact same bunch of errors. It seems to sometimes happen when i leave my mac idle but has also happened randomly whilst in use. Sometimes I cannot even spin up my containers again without restarting docker although usually i can simply docker-compose kill and then docker-compose up will work again.

roeman · June 22, 2016, 1:13pm

Hmm… related behavior here on 1.12.0-rc2-beta16 (build: 9493)

Docker compose launched nginx, php-fpm, mysql, ssh-agent containers, received the following truncated output during a benchmark (you can see mysql crash).

Exception in thread Thread-9:
Traceback (most recent call last):
File “threading.py”, line 810, in __bootstrap_inner
File “threading.py”, line 763, in run
File “compose/cli/log_printer.py”, line 190, in watch_events
File “compose/project.py”, line 343, in events
File “site-packages/docker/client.py”, line 233, in _stream_helper
File “json/init.py”, line 338, in loads
File “json/decoder.py”, line 369, in decode
ValueError: Extra data: line 2 column 1 - line 5 column 1 (char 658 - 2598)
wp_1 | [21-Jun-2016 22:12:08] NOTICE: Terminating …
db_1 | 160621 22:12:08 [Note] mysqld: Normal shutdown
db_1 |
db_1 | 160621 22:12:08 [Note] Event Scheduler: Purging the queue. 0 events
db_1 | 160621 22:12:08 InnoDB: Starting shutdown…

wp_1 | [21-Jun-2016 22:12:08] NOTICE: exiting, bye-bye!
db_1 | 160621 22:12:09 InnoDB: Error: Write to file ./ibdata1 failed at offset 0 1048576.
db_1 | InnoDB: 16384 bytes should have been written, only -1 were written.
db_1 | InnoDB: Operating system error number 107.
db_1 | InnoDB: Check that your OS and file system support files of this size.
db_1 | InnoDB: Check also that the disk is not full or a disk quota exceeded.
db_1 | InnoDB: Error number 107 means ‘Transport endpoint is not connected’.
db_1 | InnoDB: Some operating system error numbers are described at
db_1 | InnoDB: removed so I can post sans links
db_1 | 160621 22:12:09 InnoDB: Assertion failure in thread 140631388018432 in file fil0fil.c line 4661
db_1 | InnoDB: Failing assertion: ret
db_1 | InnoDB: We intentionally generate a memory trap.
db_1 | InnoDB: Submit a detailed bug report to http://bugs.mysql.com.
db_1 | InnoDB: If you get repeated assertion failures or crashes, even
db_1 | InnoDB: immediately after the mysqld startup, there may be
db_1 | InnoDB: corruption in the InnoDB tablespace. Please refer to
db_1 | InnoDB: removed so I can post sans links
db_1 | InnoDB: about forcing recovery.
db_1 | 22:12:09 UTC - mysqld got signal 6 ;
db_1 | This could be because you hit a bug. It is also possible that this binary
db_1 | or one of the libraries it was linked against is corrupt, improperly built,
db_1 | or misconfigured. This error can also be caused by malfunctioning hardware.
db_1 | We will try our best to scrape up some info that will hopefully help
db_1 | diagnose the problem, but since we have already crashed,
db_1 | something is definitely wrong and this may fail.
db_1 |
db_1 | key_buffer_size=8388608
db_1 | read_buffer_size=131072
db_1 | max_used_connections=5
db_1 | max_threads=151
db_1 | thread_count=0
db_1 | connection_count=0
db_1 | It is possible that mysqld could use up to
db_1 | key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 338512 K bytes of memory
db_1 | Hope that’s ok; if not, decrease some variables in the equation.
db_1 |
db_1 | Thread pointer: 0x0
db_1 | Attempting backtrace. You can use the following information to find out
db_1 | where mysqld died. If you see no messages after this, something went
db_1 | terribly wrong…
db_1 | stack_bottom = 0 thread_stack 0x40000
db_1 | mysqld(my_print_stacktrace+0x35)[0x7a94f5]
db_1 | mysqld(handle_fatal_signal+0x403)[0x676593]
db_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0xf8d0)[0x7fe760d5e8d0]
db_1 | /lib/x86_64-linux-gnu/libc.so.6(gsignal+0x37)[0x7fe75fe93067]
db_1 | /lib/x86_64-linux-gnu/libc.so.6(abort+0x148)[0x7fe75fe94448]
db_1 | mysqld[0x867706]
db_1 | mysqld[0x837d83]
db_1 | mysqld[0x839bf1]
db_1 | mysqld[0x839ea7]
db_1 | mysqld[0x7ecad4]
db_1 | /lib/x86_64-linux-gnu/libpthread.so.0(+0x80a4)[0x7fe760d570a4]
db_1 | /lib/x86_64-linux-gnu/libc.so.6(clone+0x6d)[0x7fe75ff4687d]
db_1 | The manual page at removed so I can post sans links contains
db_1 | information that should help you find out what is causing the crash.
Exception in thread Thread-7:
Traceback (most recent call last):
File “threading.py”, line 810, in __bootstrap_inner
File “threading.py”, line 763, in run
File “compose/cli/log_printer.py”, line 149, in tail_container_logs
File “compose/cli/log_printer.py”, line 179, in wait_on_exit
File “compose/container.py”, line 239, in wait
File “site-packages/docker/utils/decorators.py”, line 21, in wrapped
File “site-packages/docker/api/container.py”, line 439, in wait
File “site-packages/docker/client.py”, line 154, in _raise_for_status
APIError: 502 Server Error: Bad Gateway (“Bad response from Docker engine”)

danielpcox · June 22, 2016, 1:55pm

I have a similar problem, I think, on 1.12.0-rc2-beta16. I’m running a few containers with Docker Compose, one of them a web server, and when I hard-refresh a page in the browser connected to one of the exposed ports, the whole thing crashes with:

Exception in thread Thread-10:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/danielpcox/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

Exception in thread Thread-9:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/danielpcox/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")
Exception in thread Thread-7:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/danielpcox/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

Exception in thread Thread-8:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/danielpcox/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

Exception in thread Thread-6:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/danielpcox/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

aaronjensen · June 27, 2016, 8:19pm

Same thing for me, with 1.12.0-rc2-beta16 a hard refresh crashes the containers. I can reproduce it when running the container in isolation, docker-compose isn’t necessary.

Exception in thread Thread-6:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/aaronjensen/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

Exception in thread Thread-7:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/aaronjensen/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")
Exception in thread Thread-8:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/aaronjensen/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")
Exception in thread Thread-5:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 439, in wait
  File "site-packages/docker/client.py", line 154, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/aaronjensen/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

aaronjensen · June 27, 2016, 11:33pm

I should also note that this only seems to happen when I’m running my site through browser-sync. If I run it normally, it doesn’t happen. I’m not sure if anyone else has browser-sync in the mix.

danielpcox · June 28, 2016, 3:45am

Ah, I do. I wonder what that’s about. Something about virtual networking and traffic being initiated in an unexpected direction perhaps?

[EDIT] Actually, I don’t have browser-sync running in this particular case, though I’ve had trouble where it existed, and there is a node app in the mix in this case.

aaronjensen · June 28, 2016, 4:37am

Can you narrow it down to a shareable repro? It may help track it down.

toddpi314 · July 4, 2016, 7:50pm

Seeing this issue on 1.12.0-rc2 on el capitan with no relationship to browser-sync

In my particular case, this occurs seemingly randomly amid heavy TCP activity with a container.

Also note that the issue hoses docker itself. When it occurs, all containers throw the same error and everything goes down in flames.

v9chris · July 6, 2016, 2:20pm

Still seeing this issue.

Every single time my mac goes on standby docker crashes all containers and they cannot be killed or restarted without force quitting docker and restarting it. (The restart button in the docker dropdown seems to do nothing.)

Edit: Not the only time I get this. Sometimes it just happens whilst I’m working. Possibly two issues with the same result.

dionjwa · July 18, 2016, 7:18pm

I’m also seeing this, when a container comes under heavy load serving data via HTTP requests.

vtvetrivel · July 22, 2016, 1:55pm

I see this issue very often when using livereload.

boelter · July 22, 2016, 5:36pm

I’m experiencing similar issues, but I’m still trying to consistently reproduce. In my case it’s while restoring a postgres database to a data volume. I can usually get pg_restore --jobs 4 to crash with a segmentation fault, and occasionally get the thread exceptions. Very inconsistent, so I’m hesitant to link the two, but thought I’d mention it in case it sounds familiar to others. Please let me know if any specific details would be helpful. I’ll post back if I can more reliably reproduce.

emiliorodo · August 17, 2016, 2:01pm

Seeing the same thing:

^CGracefully stopping... (press Ctrl+C again to force)
...
Exception in thread Thread-11:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 190, in watch_events
  File "compose/project.py", line 343, in events

Happens when I try do ctrl-c after a docker-compose up. Tends to occur after I’ve closed my macbook and reopened it after some time. What is annoying is that a docker-compose down does clean up the containers, but Kitematic seems to be thrown off and still shows them as running.

justinbull · August 25, 2016, 3:00pm

Getting this error on the stable branch of Docker for Mac:

Version 1.12.0-a (build: 11213)
ad6ab836187e4111082447b7c0a6a74d01929a5c

Mac OS X: El Capitan 10.11.16

Starting up a bunch of micro services, around 26, for development of our front-end system.

$ docker-compose build #succeeds
foobar uses an image, skipping
...
Building nginx
...
Successfully built 4756ebaef52d
$ docker-compose up -d # appears to succeed
Starting serverapps_foobar_1
Creating serverapps_nginx_1
...

Having all of the containers boot, they all exit as 1. So I immediately launch logs -f to follow why it all falls down:

$ docker-compose logs -f
Attaching to serverapps_nginx_1, serverapps_foobar_1 # etc.
# ... logs happen here, as expected. Then suddenly (3x):
Exception in thread Thread-11:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 441, in wait
  File "site-packages/docker/client.py", line 174, in _raise_for_status
APIError: 502 Server Error: Bad Gateway ("Bad response from Docker engine")
# Followed by (~20x):
Exception in thread Thread-4:
Traceback (most recent call last):
  File "threading.py", line 810, in __bootstrap_inner
  File "threading.py", line 763, in run
  File "compose/cli/log_printer.py", line 149, in tail_container_logs
  File "compose/cli/log_printer.py", line 179, in wait_on_exit
  File "compose/container.py", line 239, in wait
  File "site-packages/docker/utils/decorators.py", line 21, in wrapped
  File "site-packages/docker/api/container.py", line 441, in wait
  File "site-packages/docker/client.py", line 174, in _raise_for_status
APIError: 500 Server Error: Internal Server Error ("dial unix /Users/jbull/Library/Containers/com.docker.docker/Data/*00000003.00000948: connect: connection refused")

At this point, doing docker-compose ps shows all entry states are Exit 1

I’m not even remotely sure what the root cause is. According to other developers who prefer the Docker Toolbox have no issues with the same repo.

Opened an issue on GitHub: https://github.com/docker/for-mac/issues/423

tzvikam · September 21, 2016, 10:53am

I have the same exact issue.
Ever since I’m working with more dockers in my docker-compose (7 right now) I get this errors constantly.
mac os: 10.11.5
Docker for Mac: 1.12.1-beta26.1 (build: 12100)

dougdoug · September 22, 2016, 2:39pm

Getting the same thing, over and over. Docker for mac completely unusable atm

I should add, mine has nothing to do with heavy load, standby mode, etc. I docker-compose up, 7 containers launch, and within 2 minutes docker fails. I’ve tried bumping my ram up to 6gb per someone’s suggestion, no luck.

jbomb · October 6, 2016, 8:22am

Same here I’ve got the same issues. Quite hard to work atm with crashes every 2 minutes.

Docker for Mac: 1.12.2-rc1-beta27 (build: 12496)
MacOs: 10.11.6

pjar · October 26, 2016, 10:14am

Same here - spent hours on setting up docker which is now stopping me to work because of this error.

UPDATE: I reduced number of processors for docker to 1 - no sign of this error since then (but that’s only for 1 day now)