Docker not responding to any commands, systemctl status says Deactivating since 17h

akshayrathod · January 29, 2020, 8:06am

Hi there,

I have a machine provisioned running Ubuntu 18.04. I had two services - API & UI running in two separate containers for this setup and while using them, the server stopped responding. Investigating the issue further, I noticed that the container running the API did not receive any request from the UI, and neither the container running the UI is giving any response.

I’m using docker-compose for spinning up these two containers, so, I was able to see the application logs.

In an attempt to fix the issue, I tried to stop the containers using docker stop <container_id> that’s when I saw this issue. The docker command is not sending any response and it just freezes even while executing something simple like docker --version

Below I have attached some logs which can help debug the issue. An important one is the status of the docker service which says Active: deactivating (stop-sigterm) since Tue 2020-01-28 13:37:04 UTC; 17h ago

There may be ways definitely to restart the host or reload daemon, but I’m more importantly seeking for the root cause of this issue and yes, some help on how to get this running again also. Help will be much appreciated.
I have tried a couple of ways to restart the service, but they didn’t work.

Thank you!

ubuntu@awesome-server:~$ sudo systemctl status docker.service 
● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: deactivating (stop-sigterm) since Tue 2020-01-28 13:37:04 UTC; 17h ago
     Docs: https://docs.docker.com
 Main PID: 13538 (dockerd)
    Tasks: 10
   CGroup: /system.slice/docker.service
           └─13538 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691192801Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/contai
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691214250Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.701614940Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344374035Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344428247Z" level=warning msg="Your kernel does not support cgroup rt period"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344441338Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344452211Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344462571Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344823400Z" level=info msg="Loading containers: start."
Jan 28 13:38:10 awesome-server dockerd[13538]: time="2020-01-28T13:38:10.313301293Z" level=info msg="Processing signal 'terminated'"

ubuntu@awesome-server:~$ sudo systemctl status docker.socket 
● docker.socket - Docker Socket for the API
   Loaded: loaded (/lib/systemd/system/docker.socket; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2020-01-28 13:37:19 UTC; 17h ago
   Listen: /var/run/docker.sock (Stream)
    Tasks: 0 (limit: 4915)
   CGroup: /system.slice/docker.socket

Jan 28 13:37:19 awesome-server systemd[1]: Starting Docker Socket for the API.
Jan 28 13:37:19 awesome-server systemd[1]: Listening on Docker Socket for the API.

ubuntu@awesome-server:~$ journalctl -u docker -e

Jan 28 13:32:34 awesome-server dockerd[4592]: time="2020-01-28T13:32:34.474241959Z" level=info msg="Container 68fa8b04a55be3e1e8fa999e78f453ccb5faed098ee3e3cc49c7f
Jan 28 13:37:04 awesome-server systemd[1]: Stopping Docker Application Container Engine...
Jan 28 13:37:04 awesome-server dockerd[4592]: time="2020-01-28T13:37:04.438044611Z" level=info msg="Processing signal 'terminated'"
Jan 28 13:37:04 awesome-server dockerd[4592]: time="2020-01-28T13:37:04.634562280Z" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/task
Jan 28 13:37:19 awesome-server dockerd[4592]: time="2020-01-28T13:37:19.438595752Z" level=error msg="Force shutdown daemon"
Jan 28 13:37:19 awesome-server dockerd[4592]: time="2020-01-28T13:37:19.438709299Z" level=info msg="Daemon shutdown complete"
Jan 28 13:37:19 awesome-server systemd[1]: Stopped Docker Application Container Engine.
Jan 28 13:37:19 awesome-server systemd[1]: Starting Docker Application Container Engine...
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.681861377Z" level=info msg="Starting up"
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.683174716Z" level=info msg="detected 127.0.0.53 nameserver, assuming systemd-resolved, so 
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.688204167Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.688275781Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" mo
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.688319686Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/contai
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.688369330Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691113654Z" level=info msg="parsed scheme: \"unix\"" module=grpc
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691154210Z" level=info msg="scheme \"unix\" not registered, fallback to default scheme" mo
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691192801Z" level=info msg="ccResolverWrapper: sending update to cc: {[{unix:///run/contai
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.691214250Z" level=info msg="ClientConn switching balancer to \"pick_first\"" module=grpc
Jan 28 13:37:19 awesome-server dockerd[13538]: time="2020-01-28T13:37:19.701614940Z" level=info msg="[graphdriver] using prior storage driver: overlay2"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344374035Z" level=warning msg="Your kernel does not support swap memory limit"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344428247Z" level=warning msg="Your kernel does not support cgroup rt period"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344441338Z" level=warning msg="Your kernel does not support cgroup rt runtime"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344452211Z" level=warning msg="Your kernel does not support cgroup blkio weight"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344462571Z" level=warning msg="Your kernel does not support cgroup blkio weight_device"
Jan 28 13:37:20 awesome-server dockerd[13538]: time="2020-01-28T13:37:20.344823400Z" level=info msg="Loading containers: start."
Jan 28 13:38:10 awesome-server dockerd[13538]: time="2020-01-28T13:38:10.313301293Z" level=info msg="Processing signal 'terminated'"

akshayfinney · February 19, 2020, 2:31am

Seeing the same issue. Any one have a resolution ?

mahmoudbatman · December 4, 2020, 10:47am

Did you find a solution I have the same problem

karmavil · February 22, 2021, 7:02am

Are you running your machine in a cluster?
If you are not declaring resource usage limits which would explain those kernel messages, make sure that the services are not having errors or loops consuming too many resources (maybe for requests).
I would suggest you test your services isolated using different networks… you know ping the service, log the requests… I don’t know it looks like it’s related with your host managing those resources (I’m aware that sounds obvious).

Topic		Replies	Views
Docker deactivating (stop-sigterm) General docker	0	1108	March 7, 2022
Docker commands are not responsive - cannot kill dockerd General	1	568	November 11, 2022
Not able to start Docker Application Container Engine General docker	0	7525	March 5, 2018
Docker commands with absolutely no response General	1	587	June 10, 2022
Docker container unresponsive, cannot be stopped, only restarting Docker Desktop helps Docker Desktop windows	6	5599	November 14, 2024

Docker not responding to any commands, systemctl status says Deactivating since 17h

Related topics