Full docker breakdown on elastic beanstalk

Hi all,

I am completely stuck with Docker on Elastic Beanstalk, and so far everything points to Docker as the reason. We deploy a HIGHLY CPU intensive single container app on EB, and after sending one request the system becomes unresponsive. That is not bad, but after a reboot, docker is broken. It is so broken that I seem to have to rebuild the system (or rm everything under /var/lib/docker) to get it up again.

This is what happens:

  • Send request, system becomes unresponsive
  • Reboot system
  • SSH into system
  • docker ps -> docker blocks, CTRL-C needed
  • docker pull busybox -> docker blocks, CTRL-C needed
  • docker info -> docker blocks, CTRL-C needed
  • the process is running and seems to be fine
  • restart docker (which seems to work)
  • repeat -> docker blocks, CTRL-C needed.

ps | grep dock gives me this:

root      4216  0.0  2.7 237156 28324 pts/2    Sl   13:54   0:00 /usr/bin/dockerd --insecure-registry hub.vpc.local --storage-driver devicemapper --storage-opt dm.thinpooldev=/dev/mapper/docker-docker--pool --storage-opt dm.use_deferred_removal=true --storage-opt dm.use_deferred_deletion=true --storage-opt dm.fs=ext4 --storage-opt dm.basesize=100G

When I look in the logs, I see this (every time I do service docker restart):

time="2017-06-08T13:54:45.848376741Z" level=info msg="libcontainerd: new containerd process, pid: 4222" 
time="2017-06-08T13:54:45.854673176Z" level=warning msg="containerd: low RLIMIT_NOFILE changing to max" current=1024 max=4096 
time="2017-06-08T13:54:45.855132313Z" level=fatal msg="open /var/run/docker/libcontainerd/containerd/12fc1d891bb1ce0b12fa43bb774690631dcf400356d8660e20cfe0c2aa59ba5b/state.json: no such file or directory" 
time="2017-06-08T13:54:51.854434881Z" level=info msg="libcontainerd: new containerd process, pid: 4229" 
time="2017-06-08T13:54:51.858216445Z" level=warning msg="containerd: low RLIMIT_NOFILE changing to max" current=1024 max=4096 
time="2017-06-08T13:54:51.858662663Z" level=fatal msg="open /var/run/docker/libcontainerd/containerd/12fc1d891bb1ce0b12fa43bb774690631dcf400356d8660e20cfe0c2aa59ba5b/state.json: no such file or directory" 

The state file is indeed not present, but the directory in which it is supposed to be is.

The docker version we use is 1.12.6.

can anybody help me out with a couple of hints here? This is literally all I got on information, and I am stuck, and this is kinda unfortunate.