Docker Community Forums

Share and learn in the Docker community.

Mysterious docker engine shutdown

I run a streaming data application in a docker container using docker engine on Ubuntu. I woke up this morning to find that the docker daemon had exited mysteriously. I ran sudo journalctl -n 50 -fu docker.service and noticed the following:

Dec 01 06:15:58 ip-172-31-11-114 systemd[1]: Stopping Docker Application Container Engine…
Dec 01 06:15:58 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:15:58.650662973Z” level=info msg=“Processing signal ‘terminated’”
Dec 01 06:16:08 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:08.741278497Z” level=info msg=“Container 675b0cd325014b323d723b688f779b8fc7f3ed535b9fc74e422ea74aa49e
3653 failed to exit within 10 seconds of signal 15 - using the force”
Dec 01 06:16:08 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:08.741278833Z” level=info msg=“Container b9f3fc1968d07eeaa5a5cd9582e76e3dabad36e5227c6654d084c18fdd8f
2025 failed to exit within 10 seconds of signal 15 - using the force”
Dec 01 06:16:09 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:09.282690931Z” level=info msg=“ignoring event” module=libcontainerd namespace=moby topic=/tasks/delet
e type="*events.TaskDelete"
Dec 01 06:16:13 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:13.653494622Z” level=error msg=“Force shutdown daemon”
Dec 01 06:16:13 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:13.653568627Z” level=info msg=“Daemon shutdown complete”
Dec 01 06:16:13 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:13.653635545Z” level=info msg=“stopping event stream following graceful shutdown” error=“context canc
eled” module=libcontainerd namespace=plugins.moby
Dec 01 06:16:13 ip-172-31-11-114 dockerd[1995]: time=“2020-12-01T06:16:13.653664594Z” level=info msg=“stopping event stream following graceful shutdown” error=“context canc
eled” module=libcontainerd namespace=moby
Dec 01 06:16:13 ip-172-31-11-114 systemd[1]: Stopped Docker Application Container Engine.

This is running on an EC2 instance on AWS. Is there anyway to dig up more information about what triggered this event? To my knowledge, nobody logged in at that the time to shutdown the service.

Thanks!

I’ve had the same thing happening on all my servers running Ubuntu Focal with unattended-upgrades enabled.

The cause was that containerd was upgraded automatically by the apt-daily-upgrade.timer job. In my instance the upgrade was from version 1.3.3-0ubuntu2 to 1.3.3.0ubuntu2.1. Docker daemon was shut down and not restarted. I had to start it manually.

This seems to be a serious mistake but on Ubuntu’s side rather than Docker’s.

Edit: There are further reports about this issue in containerd’s bugtracker with reports from April and yesterday.