So the docker service that is running on our below machine
NAME="Red Hat Enterprise Linux Server"
VERSION="7.5 (Maipo)"
ID="rhel"
restarts randomly and we are not able to figure the root cause. Below are logs of the docker service when it was restarted.
Oct 10 11:49:57 ilgss0255 dockerd-current[54807]: {"level":"error","ts":1570697397.1965716,"caller":"logger/logger.go:99","msg":"error creating symlink","container":"mapper","pod":"newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j","namespace":"fission-function","error":"symlink /var/log/containers/newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j_fission-function_mapper-20e6c7b83f6db1647de534a9d2aaf65a1640995caadf691b7652496b0b2fbc4c.log /var/log/fission/newdeploy-mapper-default-ocz1jye8-578d946c69-ksh9j_fission-function_mapper-20e6c7b83f6db1647de534a9d2aaf65a1640995caadf691b7652496b0b2fbc4c.log: file exists","stacktrace":"github.com/fission/fission/logger.createLogSymlinks\n\t/go/src/github.com/fission/fission/logger/logger.go:99\ngithub.com/fission/fission/logger.makePodLoggerController.func2\n\t/go/src/github.com/fission/fission/logger/logger.go:67\nk8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:202\nk8s.io/client-go/tools/cache.NewInformer.func1\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:309\nk8s.io/client-go/tools/cache.(*DeltaFIFO).Pop\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/delta_fifo.go:444\nk8s.io/client-go/tools/cache.(*controller).processLoop\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:150\nk8s.io/client-go/tools/cache.(*controller).processLoop-fm\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:124\nk8s.io/apimachinery/pkg/util/wait.JitterUntil.func1\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:133\nk8s.io/apimachinery/pkg/util/wait.JitterUntil\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:134\nk8s.io/apimachinery/pkg/util/wait.Until\n\t/go/pkg/mod/k8s.io/apimachinery@v0.0.0-20190116203031-d49e237a2683/pkg/util/wait/wait.go:88\nk8s.io/client-go/tools/cache.(*controller).Run\n\t/go/pkg/mod/k8s.io/client-go@v7.0.0+incompatible/tools/cache/controller.go:124\ngithub.com/fission/fission/logger.Start\n\t/go/src/github.com/fission/fission/logger/logger.go:187\nmain.runLogger\n\tsrc/github.com/fission/fission/fission-bundle/main.go:82\nmain.main\n\tsrc/github.com/fission/fission/fission-bundle/main.go:260\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:201"}
.
.
.
Oct 10 11:50:02 ilgss0255 systemd[1]: Stopping Docker Application Container Engine...
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated
Oct 10 11:50:02 ilgss0255 dockerd-current[54807]: Shutting down, got signal: Terminated
so it somehow gets restarted and we are not able to get any useful logs on how we can go about that.
Below are the resource usage of the machine at the time when docker went down
top - 11:50:01 up 27 days, 19:09, 5 users, load average: 3.47, 3.73, 4.47
Tasks: 791 total, 1 running, 789 sleeping, 0 stopped, 1 zombie
%Cpu(s): 9.2 us, 17.6 sy, 1.2 ni, 59.7 id, 11.9 wa, 0.0 hi, 0.5 si, 0.0 st
KiB Mem : 32887468 total, 1722696 free, 15673524 used, 15491248 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 11929348 avail Mem
------------------------------
top - 11:50:04 up 27 days, 19:09, 5 users, load average: 3.47, 3.73, 4.47
Tasks: 697 total, 1 running, 695 sleeping, 0 stopped, 1 zombie
%Cpu(s): 30.6 us, 35.1 sy, 0.5 ni, 22.5 id, 10.5 wa, 0.0 hi, 0.8 si, 0.0 st
KiB Mem : 32887468 total, 4241620 free, 13174436 used, 15471412 buff/cache
KiB Swap: 0 total, 0 free, 0 used. 14436464 avail Mem
below is the docker version that we are running
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64
Go version: go1.10.8
Git commit: b2f74b2/1.13.1
Built: Tue Apr 2 21:01:07 2019
OS/Arch: linux/amd64
Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-96.gitb2f74b2.el7.x86_64
Go version: go1.10.8
Git commit: b2f74b2/1.13.1
Built: Tue Apr 2 21:01:07 2019
OS/Arch: linux/amd64
Experimental: false
any pointers on how can we go ahead about debugging this would be helpful.