What to do when all docker commands hang?

Docker for AWS
1

Steps to reproduce the behavior

Running Docker 1.12.6

  1. Run a docker build on an EC2 instance (let’s say the docker build hangs)
  2. Reboot EC2 instance

After the instance reboots, whenever I try to run ANY docker command (e.g., docker info), it just hangs and doesn’t return.

I tried to restart docker, but same problem. I tried to uninstall docker (yum remove docker) and remove /var/lib/docker and reinstall, but even then it still hangs. I have to delete the EC2 instance and start all over again. How do I fix this?

2

Can you please open an issue with as many details on the system and on how to reproduce on the Docker repo?

3

A better place for a Docker for AWS issue is probably https://github.com/docker/for-aws

Please make sure to include docker-diagnose output

Thanks!

4

Please follow the below steps to solve this issue

Step 1 : Uninstall docker “sudo yum remove docker"
Step 2 : remove all docker folder
"sudo rm -rf /var/lib/docker”
“sudo rm -rf /var/run/docker”
“sudo rm /var/run/docker.*” (remove docker.sock, docker.pid files)

Step 3 : Reinstall docker "sudo yum install docker"
Step 4 : Start docker “sudo service docker start”

2 Likes
5

You don’t need to reinstall or delete the container image data. Simply do below :slight_smile: slight_smile:

Stop the service first

sudo service docker stop

Clean up some of the files as mentioned in above post from Sam.

sudo rm -rf /var/run/docker
sudo rm /var/run/docker.*"

Start service now

sudo service docker start

Start your docker image

docker start <container-name>

You will receive an error when you run the docker run at first try:

Error response from daemon: invalid header field value "oci runtime error: container with id exists: 7a244b8f5d07081538042ff64aebfe11fac1a36731526e77be53db7d94dca44d\n"
Error: failed to start containers:

Try running docker start command again. You will have your container up and running magically without any errors.

3 Likes
6

What would you suggest if

sudo service docker stop

Hangs? Can i go ahead delete those files? And is everything self there to remove?

7

If service command hangs than you should kill the PID and delete the files.
After you can try to restart using service command.

8

I KILLED docker via the system monitor, and removed said files, but sudo service docker start produces absolutely not output and the problem persists

9

This worked for me. Thanks

10

Thank you, really helpful. In case someone is using Alpine Linux like me: apk del docker instead of yum remove docker

11

I had docker stuck in the same way, but for me the solution of @tmcconechy did not work.

because my Linux did not recognize the service command.

So I did a ps followed by grep (In the following I abbreviated the output)

BA92-C02VP224HTDF:Ampersand stefjoosten$ ps -A | grep docker
30249 ??         0:01.26 /Applications/Visual Studio Code.app/Contents/Fram...
62743 ??         0:00.01 /Library/PrivilegedHelperTools/com.docker.vmnetd
62753 ??         0:00.07 /Applications/Docker.app/Contents/MacOS/com.dock...
62754 ??         0:00.06 com.docker.osxfs serve --address fd:3 --connect vm...
62755 ??         0:34.23 com.docker.vpnkit --ethernet fd:3 --port vpnkit.p...
62756 ??         0:39.43 com.docker.driver.amd64-linux -addr fd:3 -debug
62761 ??        14:28.52 com.docker.hyperkit -A -u -F vms/0/hyperkit.pi...
76926 ttys004    0:00.00 grep docker
BA92-C02VP224HTDF:Ampersand stefjoosten$

Then I selected the right processes and killed them

kill -9 30249 62743 62753 62754 62755 62756 62761

Then docker came back up again, asking a few questions along the way. And everything was fine.

12

This didn’t work for me. Both sudo service docker start and sudo service docker stop hang, regardless if I kill the PID and try again.

Cleaning up and restarting docker has had no effect. Is the only option to delete Docker at this point?

Thanks

13

I had a similar problem, docker commands would hang. I had modified /etc/docker/daemon.json to specify host information (related) so now e.g. docker info could no longer find the daemon.

Fix was simply to set DOCKER_HOST environment variable to point to the same address e.g.:

export DOCKER_HOST="tcp://127.0.0.1:2375"
1 Like
14

Was able to kill the service only with the system monitor but deleting these files and restarting the service got it working again, thanks!

15

@korayhk thanks a ton. Worked awesome.

I was running around this issue for a round a week. Most of the solutions found online proposed to delete the container. But I had a project code in the container. So I need to recover docker without deleting the container. You saved my day.

16

I was facing a similar issue and the mentioned steps did not works for me, my issue was removing /var/lib/docker/ and /var/run/docker/ I was getting some errors looking like :

rm: cannot remove /var/lib/docker/some/sub/path/ Device or resource busy

solution was to unmount these device and I could then properly remove the folders.

17

Would this delete all docker images too?!
if so, is there a way to keep docker images from being deleted while reinstalling docker ?

18

This solved my issue.

19

For me even sudo service docker stop hangs and no output!!

20

Just to add an observation I made in the past:

I used to run a (volume) plugin that was implemented as docker container. After a reboot, the docker engine wouldn’t start as the plugin was not available, and the plugin container was not started because the docker engine didn’t finish starting. After some minutes the plugin was skipped and the docker engine started, and the plugin container could be started… it was a terrible circular dependency.

Check sudo systemctl status docker while docker is hanging. If it’s caused by a plugin, you should see errors in the output.