Docker Community Forums

Share and learn in the Docker community.

Process refuses to die in container

(Pato) #1

Hello there docker gurus and fellow docker enthusiasts,
I have a small issue and thus far couldn’t find out the cause/resolution to it.

I’m trying to use a pre-built image, namely vistalab/mcr-81. This image contains a matlab compiler runtime v8.1 (MCR) on which you can execute your matlab compiled binaries the same way you’d execute your java binaries on a JRE. I tried to run a small test matlab binary that should print Hello world and die like this:
docker run -it --rm -v <PATH_TO_MY_MATLAB_BINARY>:/execute vistalab/mcr-81 /execute/<MY_MATLAB_BINARY>

As you can see this should be a fairly straightforward, single process run-it-and-die container…except it refuses to die. It does print the “Hello world”, but hangs upon trying to exit. The container is listed as running, but trying to docker exec with bash inside gives me “permission denied”. Trying to docker stop it results in my terminal session getting hanged as well.
htop shows this process as a zombie with a running child process, which consumes next to 100% processor time and heats my laptop :slightly_smiling: :

(Note: There are two zombie processes in the screenshot because I started two docker containers.)

kill and kill -9 didn’t work on the zombie, neither on its child process.
Restarting the docker daemon doesn’t help either - The conatiner is no longer shown as running, but the “Test” process tree is adopted by the init process and continues running at full heat.
The only thing that helps is restarting the computer.

I tried building my own image with MCR, using different ubuntus: vanilla trusty, trusty-debootstrap, phusion’s baseimage with myinit - nothing helped. Seems it’s not about the PID 1 zombie reaping problem, because it is the PID 1 turning zombie.

What bothers me most, of course, is that when I install MCR directly on a virtual machine and run the Test matlbab binary, it works just fine.

I’m using docker 1.10.1 on ubuntu 14.04, 64bit

I’ve pretty much exhausted my (admittedly quite limited) linux troubleshooting skills but couldn’t find out why this happens, so if anyone feels like getting a challenge, feel free to help - I can send you the small test matlab binary I compiled for you to reproduce.

Any help will be much appreciated.

(Noamguy) #2

What does “docker logs” show?

(Pato) #3

Hi there, I had given up on trying, but decided to see what docker logs show today and miraculously everything worked like a charm.
The only thing I’ve changed on my laptop is the system updates on my ubuntu, so my best guess is this has been some kind of bug that got fixed meanwhile.
@noamguy, thanks for wanting to help.

(Nathan Le Claire) #4

Sounds a lot like the infamous, might also want to look into .

(Zainlessbrombie) #5

This (the processes not being killable) is because the process is in a different namespace. While I don’t know why that means you can’t kill it, but here is how to enter its namespace:
sudo nsenter -t <PID> -i -m -u -p -n /bin/sh
Where PID is the hanging processes PID
It’s basically the same as docker exec.

Then you use ps aux to find the hanging processes PID inside the namespace and just kill -9 it