Automatic Docker Application Container Engine service stopping

General
1

This morning, I’ve found my server just had all of its containerized services stopped. Quick $ service docker status revealed that the Docker service has stopped.

The systemctl log contains the following entries:

Dec 01 06:37:23 hostname systemd[1]: Stopping Docker Application Container Engine...
Dec 01 06:37:23 hostname dockerd[1386]: time="2020-12-01T06:37:23.549668845Z" level=info msg="Processing signal 'terminated'"
Dec 01 06:37:23 hostname dockerd[1386]: time="2020-12-01T06:37:23.750915504Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:24 hostname dockerd[1386]: time="2020-12-01T06:37:24.055172364Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:24 hostname dockerd[1386]: time="2020-12-01T06:37:24.924533474Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.623422297Z" level=info msg="Container ab26fd66ac9da5797cd14dcd1b9b65e9f7a6ad3a7e782c0b15790917b1a14cb5 failed to exit within 10 seconds of
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.705585592Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.888376315Z" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Dec 01 06:37:33 hostname dockerd[1386]: time="2020-12-01T06:37:33.889285008Z" level=info msg="stopping event stream following graceful shutdown" error="context canceled" module=libcontainerd namespace=plu
Dec 01 06:37:33 hostname systemd[1]: Stopped Docker Application Container Engine.

There was no intentionally dispatched request to stop the service. How can I investigate what caused this unexpected stopping?

2

I had exactly the same happening on several of my machines, it happened pretty much at the same time as well. For me it was at (on different machines):

Dec 01 06:26:20
Dec 01 06:41:24
Dec 01 06:57:06

I have no idea what might have cause this.

Here is one log output:

Dec 01 06:41:24 systemd[1]: Stopping Docker Application Container Engine...
Dec 01 06:41:24 dockerd[642]: time="2020-12-01T06:41:24.992065100+01:00" level=info msg="Processing signal 'terminated'"
Dec 01 06:41:25 dockerd[642]: time="2020-12-01T06:41:25.331265451+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:25 dockerd[642]: time="2020-12-01T06:41:25.331841589+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.090336141+01:00" level=info msg="Container d05c12f85ba8f74646e4ab3721ae585ca128797577d78a7a03f8ba125b699175 failed to exit within 10 seconds of signal 15 - using the force"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.227778580+01:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.409600173+01:00" level=info msg="stopping event stream following graceful shutdown" error="<nil>" module=libcontainerd namespace=moby
Dec 01 06:41:35 dockerd[642]: time="2020-12-01T06:41:35.410436387+01:00" level=info msg="Daemon shutdown complete"
Dec 01 06:41:35 systemd[1]: docker.service: Succeeded.
Dec 01 06:41:35 systemd[1]: Stopped Docker Application Container Engine.
3

Ok, I think I know the reason for this. @passiday do you have unattended upgrades enabled?

See:

In my case I can see that the docker daemon was stopped after the unattended upgrade.

Dec 01 06:41:22 posthog systemd[1]: Starting Daily apt download activities...
Dec 01 06:41:23 posthog dbus-daemon[616]: [system] Activating via systemd: service name='org.freedesktop.PackageKit' unit='packagekit.service' requested by ':1.50' (uid=0 pid=1931277 comm="/usr/bin/gdbus call --system --dest org.freedeskto" label="unconfined")
Dec 01 06:41:23 posthog systemd[1]: Starting PackageKit Daemon...
Dec 01 06:41:23 posthog PackageKit[1931280]: daemon start
Dec 01 06:41:23 posthog dbus-daemon[616]: [system] Successfully activated service 'org.freedesktop.PackageKit'
Dec 01 06:41:23 posthog systemd[1]: Started PackageKit Daemon.
Dec 01 06:41:24 posthog systemd[1]: Stopping Docker Application Container Engine...

It’s just a bit weird that the daemon is not started again.

4

Same issue at approximately the same time. This happened on several Ubuntu 18.04 machines, including server-only installations. This seems like a huge failure on the part of Ubuntu :frowning:

5

Same issue at the same time. Can somebody confirm if this issue is only present on the free hub? Or Pro and Team accound were affected too ?

Thanks in advance.

6

I had the same issue at the same time as well. @boldtrn, can we be sure that the problem originates from what you’ve mentioned? Do you have any suggestion to check that and make sure?

8

What you could check are your logs. For Ubuntu you could use journalctl and scroll to the time where it happened and see what is in your logs.

I don’t have a guarantee that this is the reason, but for me it seems like it.