Docker Community Forums

Share and learn in the Docker community.

Docker service is restarting randomly and we are not able to figure out why

So the docker service that is running on our below machine

NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"

restarts randomly and we are not able to figure the root cause. Below are logs of the docker service when it was restarted.

Oct 10 11:49:57 ilgss0255 dockerd-current[54807]: {"level":"error","ts":1570697397.1965716,"caller":"logger/logger.go:99","msg":"error creating symlink","container":"mapper","pod":"newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j","namespace":"fission-function","error":"symlink /var/log/containers/newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j_fission-function_mapper-20e6c7b83f6db1647de534a9d2aaf65a1640995caadf691b7652496b0b2fbc4c.log /var/log/fission/newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j_fission-function_mapper-20e6c7b83f6db1647de534a9d2aaf65a1640995caadf691b7652496b0b2fbc4c.log: file exists","stacktrace":"github.com/fission/fission/logger.createLogSymlinks\n\t/go/src/github.com/fission/fission/logger/logger.go:99\ngithub.com/fission/fission/logger.makePodLoggerController.func2\n\t/go/src/github.com/fission/fission/logger/logger.go:67\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:202\nk8s.io/client-go/tools/cache.NewInformer.func1\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:309\nk8s.io/client-go/tools/cache.(*DeltaFIFO).Pop\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/delta_fifo.go:444\nk8s.io/client-go/tools/cache.(*controller).processLoop\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:150\nk8s.io/client-go/tools/cache.(*controller).processLoop-fm\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:124\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:88\nk8s.io/client-go/tools/cache.(*controller).Run\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:124\ngithub.com/fission/fission/logger.Start\n\t/go/src/github.com/fission/fission/logger/logger.go:187\nmain.runLogger\n\tsrc/github.com/fission/fission/fission-bundle/main.go:82\nmain.main\n\tsrc/github.com/fission/fission/fission-bundle/main.go:260\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201"}
.
.
.
Oct 10 11:50:02 ilgss0255 systemd[1]: Stopping Docker Application Container Engine...
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated 

so it somehow gets restarted and we are not able to get any useful logs on how we can go about that.
Below are the resource usage of the machine at the time when docker went down

 
top - 11:50:01 up 27 days, 19:09,  5 users,  load average: 3.47, 3.73, 4.47
Tasks: 791 total,   1 running, 789 sleeping,   0 stopped,   1 zombie
%Cpu(s):  9.2 us, 17.6 sy,  1.2 ni, 59.7 id, 11.9 wa,  0.0 hi,  0.5 si,  0.0 st
KiB Mem : 32887468 total,  1722696 free, 15673524 used, 15491248 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 11929348 avail Mem
------------------------------

top - 11:50:04 up 27 days, 19:09,  5 users,  load average: 3.47, 3.73, 4.47
Tasks: 697 total,   1 running, 695 sleeping,   0 stopped,   1 zombie
%Cpu(s): 30.6 us, 35.1 sy,  0.5 ni, 22.5 id, 10.5 wa,  0.0 hi,  0.8 si,  0.0 st
KiB Mem : 32887468 total,  4241620 free, 13174436 used, 15471412 buff/cache
KiB Swap:        0 total,        0 free,        0 used. 14436464 avail Mem

below is the docker version that we are running

Client:
 Version:         1.13.1
 API version:     1.26
 Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Tue Apr  2 21:01:07 2019
 OS/Arch:         linux/amd64

Server:
 Version:         1.13.1
 API version:     1.26 (minimum version 1.12)
 Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64
 Go version:      go1.10.8
 Git commit:      b2f74b2/1.13.1
 Built:           Tue Apr  2 21:01:07 2019
 OS/Arch:         linux/amd64
 Experimental:    false

any pointers on how can we go ahead about debugging this would be helpful.

Seems like a docker package distributed by redhat, which is maintained and supported by redhat.
I guess raising a support ticket at redhat is your best shot.

Thanks for the input, lets see if something can be done about that.