Docker Community Forums

Share and learn in the Docker community.

Automatic Docker Application Container Engine service stopping

This morning, I’ve found my server just had all of its containerized services stopped. Quick $ service docker status revealed that the Docker service has stopped.

The systemctl log contains the following entries:

Dec 01 06:37:23 hostname systemd[1]: Stopping Docker Application Container Engine...
Dec 01 06:37:23 hostname dockerd[1386]: time="2020-12-01T06:37:23.549668845Z" level=info msg="Processing signal 'terminated'"
Dec 01 06:37:23 hostname dockerd[1386]: time="2020-12-01T06:37:23.750915504Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:24 hostname dockerd[1386]: time="2020-12-01T06:37:24.055172364Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:24 hostname dockerd[1386]: time="2020-12-01T06:37:24.924533474Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.623422297Z" level=info msg="Container ab26fd66ac9da5797cd14dcd1b9b65e9f7a6ad3a7e782c0b15790917b1a14cb5 failed to exit within 10 seconds of
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.705585592Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.888376315Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.889285008Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plu
Dec 01 06:37:33 hostname systemd[1]: Stopped Docker Application Container Engine.

There was no intentionally dispatched request to stop the service. How can I investigate what caused this unexpected stopping?

2 Likes

I had exactly the same happening on several of my machines, it happened pretty much at the same time as well. For me it was at (on different machines):

Dec 01 06:26:20
Dec 01 06:41:24
Dec 01 06:57:06

I have no idea what might have cause this.

Here is one log output:

Dec 01 06:41:24 systemd[1]: Stopping Docker Application Container Engine...
Dec 01 06:41:24 dockerd[642]: time="2020-12-01T06:41:24.992065100+01:00" level=info msg="Processing signal 'terminated'"
Dec 01 06:41:25 dockerd[642]: time="2020-12-01T06:41:25.331265451+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:25 dockerd[642]: time="2020-12-01T06:41:25.331841589+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.090336141+01:00" level=info msg="Container d05c12f85ba8f74646e4ab3721ae585ca128797577d78a7a03f8ba125b699175 failed to exit within 10 seconds of signal 15 - using the force"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.227778580+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.409600173+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.410436387+01:00" level=info msg="Daemon shutdown complete"
Dec 01 06:41:35 systemd[1]: docker.service: Succeeded.
Dec 01 06:41:35 systemd[1]: Stopped Docker Application Container Engine.

Ok, I think I know the reason for this. @passiday do you have unattended upgrades enabled?

See:

In my case I can see that the docker daemon was stopped after the unattended upgrade.

Dec 01 06:41:22 posthog systemd[1]: Starting Daily apt download activities...
Dec 01 06:41:23 posthog dbus-daemon[616]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.50' (uid=0 pid=1931277 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Dec 01 06:41:23 posthog systemd[1]: Starting PackageKit Daemon...
Dec 01 06:41:23 posthog PackageKit[1931280]: daemon start
Dec 01 06:41:23 posthog dbus-daemon[616]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Dec 01 06:41:23 posthog systemd[1]: Started PackageKit Daemon.
Dec 01 06:41:24 posthog systemd[1]: Stopping Docker Application Container Engine...

It’s just a bit weird that the daemon is not started again.

Same issue at approximately the same time. This happened on several Ubuntu 18.04 machines, including server-only installations. This seems like a huge failure on the part of Ubuntu :frowning:

Same issue at the same time. Can somebody confirm if this issue is only present on the free hub? Or Pro and Team accound were affected too ?

Thanks in advance.

I had the same issue at the same time as well. @boldtrn, can we be sure that the problem originates from what you’ve mentioned? Do you have any suggestion to check that and make sure?

I had a similar issue. This was how I fixed it permanently:

Delete everything in /var/lib/docker. This will delete exiting container and images:

rm -rf /var/lib/docker
Then configure your daemon to use the “overlay” storage driver. Set the following flags in the daemon.json file located in “/etc/docker/”. If the file doesn’t exist, just create it, and add the content below:

{
“graph”: “/mnt/docker-data”,
“storage-driver”: “overlay”
}
Now start Docker normally again and all should work fine and always.

What you could check are your logs. For Ubuntu you could use journalctl and scroll to the time where it happened and see what is in your logs.

I don’t have a guarantee that this is the reason, but for me it seems like it.

Once again lewish95 the Let me Google that for you bot has committed plagiarism. (ref)