Docker swarm crash during large file copies

Docker Engine Swarm
1

We have a system of micro services running in Docker containers. During large file copies of over 25GB, the docker swarm crashes and restarts following “context deadline exceeded” error messages in the docker logs.

Can you help determine why these crashes are occurring, how to stop them, and/or how to increase the deadline timeout.

Here is information about our system followed by a summary of the captured logs (as a new forum user I was not able to attach a file or paste additional text).
You’ll notice that there was an initial round of “DeadlineExceeded” and stack trace dumps followed by many minutes of relative calm and recovery. Then, a second round of “DeadlineExceeded” and stack track logs before the swarm finally crashed. All of this occurred while trying to copy over 100GB of several hundred files. There was more than enough space on our 500GB drive at all times.

OS: Linux (custom distribution made with Yocto and OpenEmbedded)

Docker Version
docker --version
Docker version 18.09.3, build 774a1f4

docker version
Client: Docker Engine - Community
Version: 18.09.3
API version: 1.39
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:32:01 2019
OS/Arch: linux/amd64
Experimental: false

Server: Docker Engine - Community
Engine:
Version: 18.09.3
API version: 1.39 (minimum version 1.12)
Go version: go1.10.8
Git commit: 774a1f4
Built: Thu Feb 28 06:40:51 2019
OS/Arch: linux/amd64
Experimental: false

Docker Info
docker info
Containers: 50
Running: 15
Paused: 0
Stopped: 35
Images: 25
Server Version: 18.09.3
Storage Driver: btrfs
Build Version: Btrfs v4.7.3
Library Version: 101
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: krm9pgu98urqomd3hoklhfk45
Is Manager: true
ClusterID: ktrct3lmxakkfiropayqmly9p
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 127.0.0.1
Manager Addresses:
127.0.0.1:2377
Runtimes: runc rundmc
Default Runtime: rundmc
Init Binary: docker-init
containerd version: e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e
runc version: 6635b4f0c6af3810594d2770f662f34ddc15b40d
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.9.115-yocto-standard
Operating System: GNARBOX OS 2.0.6.73 (sumo)
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 3.763GiB
Name: gnarbox-2.0-G09190178
ID: COUM:OM7G:MRCU:L6Q5:QXSL:FIUP:HCYP:UH7H:T6MD:HOG5:P2FH:ENKT
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): true
File Descriptors: 229
Goroutines: 453
System Time: 2019-05-01T12:56:50.059811322-07:00
EventsListeners: 16
Username: dmanpearl
Registry: htps://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

journalctl docker output during crash:
(see “DeadlineExceeded” below and the stack trace starting with “github.com/docker/docker/pkg/signal.DumpStacks” further below)

[95779.940768] CPU0: Core temperature above threshold, cpu clock throttled (total events = 12455)
May 01 10:32:19 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:32:19.963615708-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:ivvwd8cj7k3gtmxwo0uiye5gn leaving:false netPeers:1 entries:6 Queue qLen:0 netMsg/s:0”
May 01 10:32:19 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:32:19.965376233-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:pektiiyg8xeh7wy5ncqrpzp8d leaving:false netPeers:1 entries:29 Queue qLen:0 netMsg/s:0”
[96080.551920] CPU0: Core temperature above threshold, cpu clock throttled (total events = 12639)
May 01 10:37:20 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:37:20.164858196-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:ivvwd8cj7k3gtmxwo0uiye5gn leaving:false netPeers:1 entries:6 Queue qLen:0 netMsg/s:0”
May 01 10:37:20 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:37:20.170479373-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:pektiiyg8xeh7wy5ncqrpzp8d leaving:false netPeers:1 entries:29 Queue qLen:0 netMsg/s:0”
May 01 10:42:21 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:42:21.484841147-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:ivvwd8cj7k3gtmxwo0uiye5gn leaving:false netPeers:1 entries:6 Queue qLen:0 netMsg/s:0”
May 01 10:42:21 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:42:21.485112861-07:00” level=info msg=“NetworkDB stats gnarbox-2.0-G09190178(0579bf7e8af4) - netID:pektiiyg8xeh7wy5ncqrpzp8d leaving:false netPeers:1 entries:29 Queue qLen:0 netMsg/s:0”
[96424.299196] CPU0: Core temperature above threshold, cpu clock throttled (total events = 13023)
May 01 10:42:52 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:42:45.297436763-07:00” level=error msg=“heartbeat to manager { } failed” error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” method="(*session).heartbeat" module=node/agent node.id=krm9pgu98urqomd3hoklhfk45 session.id=8r62jcn50ymsdwv9vsojo0kvc sessionID=8r62jcn50ymsdwv9vsojo0kvc
May 01 10:43:02 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:43:02.715683811-07:00” level=error msg=“agent: session failed” backoff=100ms error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:43:04 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:43:04.607136938-07:00” level=info msg=“manager selected by agent for new session: { }” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:43:05 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:43:05.262355999-07:00” level=info msg=“waiting 50.960741ms before registering session” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:43:05 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:43:05.890014267-07:00” level=info msg=“worker krm9pgu98urqomd3hoklhfk45 was successfully registered” method="(*Dispatcher).register"
May 01 10:46:08 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:46:07.999686211-07:00” level=error msg=“heartbeat to manager { } failed” error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” method="(*session).heartbeat" module=node/agent node.id=krm9pgu98urqomd3hoklhfk45 session.id=lqbmtt3eq45xck3j0mu2lff0x sessionID=lqbmtt3eq45xck3j0mu2lff0x
May 01 10:46:09 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:46:09.797886539-07:00” level=error msg=“agent: session failed” backoff=100ms error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:46:09 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:46:09.904736858-07:00” level=info msg=“manager selected by agent for new session: { }” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:46:09 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:46:09.905729115-07:00” level=info msg=“waiting 43.86176ms before registering session” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:46:30 gnarbox-2.0-G09190178 dockerd[751]: sync duration of 17.205715644s, expected less than 1s
May 01 10:47:02 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:02.578995573-07:00” level=error msg=“heartbeat to manager { } failed” error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” method="(*session).heartbeat" module=node/agent node.id=krm9pgu98urqomd3hoklhfk45 session.id=lqbmtt3eq45xck3j0mu2lff0x sessionID=lqbmtt3eq45xck3j0mu2lff0x
May 01 10:47:02 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:02.579406738-07:00” level=error msg=“agent: session failed” backoff=100ms error=“rpc error: code = DeadlineExceeded desc = context deadline exceeded” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:02 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:02.579792991-07:00” level=info msg=“manager selected by agent for new session: { }” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:02 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:02.580094131-07:00” level=info msg=“waiting 70.656634ms before registering session” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:07 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:07.651452041-07:00” level=error msg=“agent: session failed” backoff=300ms error=“session initiation timed out” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:07 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:07.652285760-07:00” level=info msg=“manager selected by agent for new session: { }” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:07 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:07.652459723-07:00” level=info msg=“waiting 147.235327ms before registering session” module=node/agent node.id=krm9pgu98urqomd3hoklhfk45
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: sync duration of 21.094740575s, expected less than 1s
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: time=“2019-05-01T10:47:09.282909097-07:00” level=error msg=“Attempting to transfer leadership” raft_id=174e8d80e3ea2ff4
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 648 [running]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/pkg/signal.DumpStacks(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/pkg/signal/trap.go:83 +0xaa
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/manager/state/raft.(*Node).Run(0xc420851180, 0x26a6240, 0xc422f8b100, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/state/raft/raft.go:597 +0x17e8
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/manager.(*Manager).Run.func6(0xc4207aaea0, 0x26a6240, 0xc420c5c800)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/manager.go:588 +0x4c
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/manager.(*Manager).Run
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/manager.go:587 +0x1544
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 1 [chan receive, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: main.(*DaemonCli).start(0xc420717800, 0xc4206bd200, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:240 +0xc80
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: main.runDaemon(0xc4206bd200, 0xc420666100, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:7 +0x47
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: main.newDaemonCommand.func1(0xc420656a00, 0xc420840a00, 0x0, 0x4, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc420656a00, 0xc42003a150, 0x4, 0x4, 0xc420656a00, 0xc42003a150)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:762 +0x46a
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc420656a00, 0x267bb40, 0x2254580, 0x267bb50)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:852 +0x30c
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc420656a00, 0xc42000e020, 0x4d067f)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:800 +0x2d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: main.main()
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/cmd/dockerd/docker.go:70 +0xa2
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 20 [syscall, 90 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os/signal.signal_recv(0x268fe00)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/sigqueue.go:139 +0xa8
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os/signal.loop()
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/signal/signal_unix.go:22 +0x24
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by os/signal.init.0
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/signal/signal_unix.go:28 +0x43
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 14 [select]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc420189540)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:144 +0x11f
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:29 +0x5a
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 30 [select]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc4207aa820, 0x26a6240, 0xc420840ac0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:252 +0x83e
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/libcontainerd/supervisor.Start
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3fa
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 32 [syscall, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: syscall.Syscall6(0xf7, 0x1, 0x30f, 0xc4200555c8, 0x1000004, 0x0, 0x0, 0x2a, 0xc4207aa820, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os.(*Process).blockUntilWaitable(0xc420662420, 0x40482b, 0xc4206bc480, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/wait_waitid.go:31 +0x9a
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os.(*Process).wait(0xc420662420, 0xc4206bc401, 0x9b4c17, 0xc420055750)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/exec_unix.go:22 +0x3e
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os.(*Process).Wait(0xc420662420, 0xc420055728, 0xc420467c00, 0x2674e30)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/exec.go:123 +0x2d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: os/exec.(*Cmd).Wait(0xc4201962c0, 0xc4200557b8, 0x9c630b)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/os/exec/exec.go:461 +0x5e
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd.func1(0xc4201962c0, 0xc4207aa820)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:208 +0x31
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:206 +0x3f8
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 33 [select, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc4201ac2a0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x184
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x41
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 66 [select, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420840ec0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x14c
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x151
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 67 [select, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4201c2000)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1373 +0x23d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc4201c2000)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:949 +0x1b7
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:940 +0xe3
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 41 [IO wait]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.runtime_pollWait(0x7fd16495ad60, 0x72, 0xc42006dbb8)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/netpoll.go:173 +0x59
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*pollDesc).wait(0xc4201e4298, 0x72, 0xffffffffffffff00, 0x26886a0, 0x32713b0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*pollDesc).waitRead(0xc4201e4298, 0xc4201ec000, 0x8000, 0x8000)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*FD).Read(0xc4201e4280, 0xc4201ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17f
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net.(*netFD).Read(0xc4201e4280, 0xc4201ec000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/net/fd_unix.go:202 +0x51
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net.(*conn).Read(0xc42016c038, 0xc4201ec000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/net/net.go:176 +0x6c
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: bufio.(*Reader).Read(0xc42083ea80, 0xc4208702d8, 0x9, 0x9, 0x20, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/bufio/bufio.go:216 +0x23a
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: io.ReadAtLeast(0x2681120, 0xc42083ea80, 0xc4208702d8, 0x9, 0x9, 0x9, 0xc42006ddf0, 0x400dd0, 0xc42006de9f)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/io/io.go:309 +0x88
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: io.ReadFull(0x2681120, 0xc42083ea80, 0xc4208702d8, 0x9, 0x9, 0x62e33, 0x62e332312e26c, 0x5cc9db9d)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/io/io.go:327 +0x5a
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc4208702d8, 0x9, 0x9, 0x2681120, 0xc42083ea80, 0x0, 0xbf2a94c700000000, 0x57f32026812b, 0x32c7920)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc4208702a0, 0xc42312e260, 0xc42312e260, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420206000)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:1123 +0x117
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0xb41
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 42 [select]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc420164400, 0x1, 0x0, 0x0, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x135
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc4201cc060)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x1be
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc420206000)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x7e
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0xc91
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 43 [select, 1611 minutes, locked to thread]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: runtime.gopark(0x2674cd0, 0x0, 0x1a6b2b3, 0x6, 0x18, 0x1)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/proc.go:291 +0x120
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: runtime.selectgo(0xc420472750, 0xc4200969c0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/select.go:392 +0xe56
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: runtime.ensureSigM.func1()
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/signal_unix.go:549 +0x1f6
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: runtime.goexit()
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 83 [chan receive, 90 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/pkg/signal.Trap.func1(0xc42083ed20, 0x2683a20, 0xc42003a230, 0xc4207a2c60)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x5d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/pkg/signal.Trap
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x120
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 55 [chan receive, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1(0xc4201d02a0, 0x1a7a122, 0xf)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:18 +0x46
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0xc1
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: goroutine 60 [IO wait, 1611 minutes]:
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.runtime_pollWait(0x7fd16495ac90, 0x72, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/netpoll.go:173 +0x59
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*pollDesc).wait(0xc42045dd18, 0x72, 0xc420189c00, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*pollDesc).waitRead(0xc42045dd18, 0xffffffffffffff00, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: internal/poll.(*FD).Accept(0xc42045dd00, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1aa
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net.(*netFD).accept(0xc42045dd00, 0xc4201cbe58, 0x411dda, 0x30)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/net/fd_unix.go:238 +0x44
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net.(*UnixListener).accept(0xc4206e4e70, 0x53810c, 0x24eee60, 0xc4207b3320)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/net/unixsock_posix.go:162 +0x34
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net.(*UnixListener).Accept(0xc4206e4e70, 0xc420042028, 0x22fb680, 0x32490f0, 0x2633d20)
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/net/unixsock.go:253 +0x4b
May 01 10:47:09 gnarbox-2.0-G09190178 dockerd[751]: net/http.(*Server).Serve(0xc4207a1040, 0x26a4480, 0xc4206e4e70, 0x0, 0x0)

May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/cmd/dockerd/daemon_unix.go:58 +0xb7
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: goroutine 260 [semacquire, 1615 minutes]:
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: sync.runtime_Semacquire(0xc420d66cfc)
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/runtime/sema.go:56 +0x3b
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: sync.(*WaitGroup).Wait(0xc420d66cf0)
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /usr/local/go/src/sync/waitgroup.go:129 +0x74
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/node.(*Node).run(0xc420a624b0, 0x26a6300, 0xc420b057d0, 0x0, 0x0)
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/node/node.go:504 +0xa7a
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: created by github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/node.(*Node).Start.func1
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/node/node.go:256 +0x8a
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: goroutine 261 [select, 1615 minutes]:
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/vendor/github.com/docker/swarmkit/node.(*Node).Err(0xc420a624b0, 0x26a6280, 0xc420042028, 0x465446, 0xc420fba260)
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/node/node.go:548 +0xf1
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: github_dot_com/docker/docker/daemon/cluster.(*nodeRunner).handleNodeExit(0xc420c02180, 0xc420a624b0)
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: /go/src/github.com/docker/docker/daemon/cluster/noderunner.go:270 +0x49
May 01 10:51:07 gnarbox-2.0-G09190178 dockerd[751]: sync duration of 1.356572079s, expected less than 1s
[96968.617763] IPVS: rr: FWM 654 0x0000028E - no destination available
[96968.796889] IPVS: rr: FWM 664 0x00000298 - no destination available
[96968.820535] IPVS: rr: FWM 654 0x0000028E - no destination available
[96968.860949] IPVS: rr: FWM 664 0x00000298 - no destination available
[96968.905092] IPVS: rr: FWM 664 0x00000298 - no destination available
[96969.097138] IPVS: rr: FWM 667 0x0000029B - no destination available
[96969.223083] IPVS: rr: FWM 654 0x0000028E - no destination available
[96969.225076] IPVS: rr: FWM 662 0x00000296 - no destination available
[96969.231309] IPVS: rr: FWM 662 0x00000296 - no destination available
[96969.353058] IPVS: rr: FWM 667 0x0000029B - no destination available

2 Likes
2

We are actually seeing this when running a regular linux cp command on the host machine outside the swarm. The swarm becomes unresponsive and services won’t restart until the copy finishes.
Docker eventually crashes downstream of the DeadlineExceeded error (see logs around 08:11:48).

May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:44.909513628-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22uzzuj9bme8kpc2go96kgymitk%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:45.247012394-07:00" level=debug msg="daemon is not responding" binary=containerd error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=libcontainerd
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:45.254579372-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22xkhuiqcol362tzdc29dm4pl58%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:52.335717521-07:00" level=debug msg="daemon is not responding" binary=containerd error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=libcontainerd
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:52.340343614-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22y60lkw1r7nbdxp2cee3o8fb1d%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:52.348905119-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22zbh2521n5yz70cfi5fctcodq8%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:49.549398019-07:00" level=error msg="agent: session failed" backoff=8s error="session initiation timed out" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:54.372891992-07:00" level=debug msg="agent: rebuild session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:54.373095532-07:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:54.373497434-07:00" level=info msg="waiting 2.256350966s before registering session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.142206457-07:00" level=debug msg="Calling GET /v1.24/services"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.459206597-07:00" level=debug msg="Calling GET /v1.24/version"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.460545826-07:00" level=debug msg="Calling GET /v1.24/networks?filters=%7B%22scope%22%3A%7B%22swarm%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.545543242-07:00" level=debug msg="Name To resolve: bluebird."
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.545617600-07:00" level=debug msg="[resolver] lookup name bluebird. present without IPv6 address"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.546024930-07:00" level=debug msg="Name To resolve: bluebird."
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.546220816-07:00" level=debug msg="[resolver] lookup for bluebird.: IP [10.0.2.25]"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:56.629899613-07:00" level=debug msg="(*session).start" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:04:57.408174155-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%2223etfv84o0z9qlbvy4wp4qcdw%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:01.630614583-07:00" level=error msg="agent: session failed" backoff=8s error="session initiation timed out" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:01.630894344-07:00" level=debug msg="agent: rebuild session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:01.631077062-07:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:01.631337802-07:00" level=info msg="waiting 5.968179746s before registering session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:02.191958582-07:00" level=debug msg="daemon is not responding" binary=containerd error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=libcontainerd
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:04.936047413-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%2273w7w46moewdr3c66n8dta12c%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:05.405928404-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%227lqrue6945jmepctg3qzgt4w9%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:06.825736202-07:00" level=debug msg="Name To resolve: bluebird."
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:06.825904189-07:00" level=debug msg="[resolver] lookup name bluebird. present without IPv6 address"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:06.826740283-07:00" level=debug msg="Name To resolve: bluebird."
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:06.826901604-07:00" level=debug msg="[resolver] lookup for bluebird.: IP [10.0.2.25]"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:07.013076018-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%228iuk9xt72oetxh74cap2kwthp%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:07.599853985-07:00" level=debug msg="(*session).start" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.226383842-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22bxsjux8dvkxzq6n4cyqexzp3m%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.229892953-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22jprq5ubbq23vpmttq1d01578f%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.600819015-07:00" level=error msg="agent: session failed" backoff=8s error="session initiation timed out" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.601064611-07:00" level=debug msg="agent: rebuild session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.601312846-07:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:12.601758980-07:00" level=info msg="waiting 1.05789663s before registering session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:13.660070842-07:00" level=debug msg="(*session).start" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:18.660983475-07:00" level=error msg="agent: session failed" backoff=8s error="session initiation timed out" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:18.661254182-07:00" level=debug msg="agent: rebuild session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:18.661530079-07:00" level=info msg="manager selected by agent for new session: { }" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:18.661780152-07:00" level=info msg="waiting 5.14479447s before registering session" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:19.371222266-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22l890tw75b36dd81b46zet3m6a%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:20.875879052-07:00" level=error msg="Attempting to transfer leadership" raft_id=2a32e986c79de921
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:27.331818207-07:00" level=debug msg="(*session).start" module=node/agent node.id=yissq3vvxx78rl6019z8erya6
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:27.431662441-07:00" level=debug msg="daemon is not responding" binary=containerd error="rpc error: code = DeadlineExceeded desc = context deadline exceeded" module=libcontainerd
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:27.457297748-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22o94v27satq55zxcvji6g87cdl%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:27.478825740-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22q5gv4tkokwoi85262dbrvszru%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:27.482263858-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22qqia6vkplvxvthdq5aq78vud8%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:28.187291973-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22r8hh45snh6xgp6cz6dn4xq4sc%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: time="2019-05-15T08:05:28.753323317-07:00" level=debug msg="Calling GET /v1.24/tasks?filters=%7B%22desired-state%22%3A%7B%22running%22%3Atrue%7D%2C%22service%22%3A%7B%22uzzuj9bme8kpc2go96kgymitk%22%3Atrue%7D%7D"
May 15 08:11:48 <hostname> dockerd[765]: goroutine 573 [running]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/pkg/signal.DumpStacks(0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/pkg/signal/trap.go:83 +0xaa
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/state/raft.(*Node).Run(0xc4205dec80, 0x26a6240, 0xc4228999c0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/state/raft/raft.go:597 +0x17e8
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/github.com/docker/swarmkit/manager.(*Manager).Run.func6(0xc42073f860, 0x26a6240, 0xc420b20480)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/manager.go:588 +0x4c
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/github.com/docker/swarmkit/manager.(*Manager).Run
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/docker/swarmkit/manager/manager.go:587 +0x1544
May 15 08:11:48 <hostname> dockerd[765]: goroutine 1 [chan receive, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: main.(*DaemonCli).start(0xc4206fb1d0, 0xc4206bcae0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/cmd/dockerd/daemon.go:240 +0xc80
May 15 08:11:48 <hostname> dockerd[765]: main.runDaemon(0xc4206bcae0, 0xc420164700, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/cmd/dockerd/docker_unix.go:7 +0x47
May 15 08:11:48 <hostname> dockerd[765]: main.newDaemonCommand.func1(0xc4206b8280, 0xc420719400, 0x0, 0x4, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:29 +0x5d
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).execute(0xc4206b8280, 0xc42003a150, 0x4, 0x4, 0xc4206b8280, 0xc42003a150)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:762 +0x46a
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4206b8280, 0x267bb40, 0x2254580, 0x267bb50)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:852 +0x30c
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4206b8280, 0xc42000e020, 0x4d067f)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/github.com/spf13/cobra/command.go:800 +0x2d
May 15 08:11:48 <hostname> dockerd[765]: main.main()
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/cmd/dockerd/docker.go:70 +0xa2
May 15 08:11:48 <hostname> dockerd[765]: goroutine 10 [syscall, 10 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: os/signal.signal_recv(0x268fe00)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/sigqueue.go:139 +0xa8
May 15 08:11:48 <hostname> dockerd[765]: os/signal.loop()
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/signal/signal_unix.go:22 +0x24
May 15 08:11:48 <hostname> dockerd[765]: created by os/signal.init.0
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/signal/signal_unix.go:28 +0x43
May 15 08:11:48 <hostname> dockerd[765]: goroutine 51 [select]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/go.opencensus.io/stats/view.(*worker).start(0xc420718640)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:144 +0x11f
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/go.opencensus.io/stats/view.init.0
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/go.opencensus.io/stats/view/worker.go:29 +0x5a
May 15 08:11:48 <hostname> dockerd[765]: goroutine 60 [select]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/libcontainerd/supervisor.(*remote).monitorDaemon(0xc42073f1e0, 0x26a6240, 0xc420719580)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:252 +0x83e
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/libcontainerd/supervisor.Start
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:90 +0x3fa
May 15 08:11:48 <hostname> dockerd[765]: goroutine 61 [syscall, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: syscall.Syscall6(0xf7, 0x1, 0x317, 0xc4207b45c8, 0x1000004, 0x0, 0x0, 0x2a, 0xc42073f1e0, 0xc4203b4b10)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/syscall/asm_linux_amd64.s:44 +0x5
May 15 08:11:48 <hostname> dockerd[765]: os.(*Process).blockUntilWaitable(0xc42078fa40, 0x40482b, 0xc4200a23c0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/wait_waitid.go:31 +0x9a
May 15 08:11:48 <hostname> dockerd[765]: os.(*Process).wait(0xc42078fa40, 0xc4200a2301, 0x9b4c17, 0xc4207b4750)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/exec_unix.go:22 +0x3e
May 15 08:11:48 <hostname> dockerd[765]: os.(*Process).Wait(0xc42078fa40, 0xc4207b4728, 0xc4205c2850, 0x2674e30)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/exec.go:123 +0x2d
May 15 08:11:48 <hostname> dockerd[765]: os/exec.(*Cmd).Wait(0xc4207949a0, 0xc4207b47b8, 0x9c630b)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/os/exec/exec.go:461 +0x5e
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd.func1(0xc4207949a0, 0xc42073f1e0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:208 +0x31
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/libcontainerd/supervisor.(*remote).startContainerd
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/libcontainerd/supervisor/remote_daemon.go:206 +0x3f8
May 15 08:11:48 <hostname> dockerd[765]: goroutine 62 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc4206fbc20)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x184
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x41
May 15 08:11:48 <hostname> dockerd[765]: goroutine 63 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420719940)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x14c
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x151
May 15 08:11:48 <hostname> dockerd[765]: goroutine 64 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc420640000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1373 +0x23d
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc420640000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:949 +0x1b7
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:940 +0xe3
May 15 08:11:48 <hostname> dockerd[765]: goroutine 67 [IO wait]:
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.runtime_pollWait(0x7f1189898d60, 0x72, 0xc42006cbb8)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/netpoll.go:173 +0x59
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*pollDesc).wait(0xc42065e618, 0x72, 0xffffffffffffff00, 0x26886a0, 0x32713b0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*pollDesc).waitRead(0xc42065e618, 0xc420196000, 0x8000, 0x8000)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*FD).Read(0xc42065e600, 0xc420196000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_unix.go:157 +0x17f
May 15 08:11:48 <hostname> dockerd[765]: net.(*netFD).Read(0xc42065e600, 0xc420196000, 0x8000, 0x8000, 0x11, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/fd_unix.go:202 +0x51
May 15 08:11:48 <hostname> dockerd[765]: net.(*conn).Read(0xc42016e018, 0xc420196000, 0x8000, 0x8000, 0x0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/net.go:176 +0x6c
May 15 08:11:48 <hostname> dockerd[765]: bufio.(*Reader).Read(0xc42064e300, 0xc420666038, 0x9, 0x9, 0x20, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/bufio/bufio.go:216 +0x23a
May 15 08:11:48 <hostname> dockerd[765]: io.ReadAtLeast(0x2681120, 0xc42064e300, 0xc420666038, 0x9, 0x9, 0x9, 0xc42006cdf0, 0x400dd0, 0xc42006ce9f)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/io/io.go:309 +0x88
May 15 08:11:48 <hostname> dockerd[765]: io.ReadFull(0x2681120, 0xc42064e300, 0xc420666038, 0x9, 0x9, 0x261da297, 0x261da29722ea1e2c, 0x5cdc2ab8)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/io/io.go:327 +0x5a
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/golang.org/x/net/http2.readFrameHeader(0xc420666038, 0x9, 0x9, 0x2681120, 0xc42064e300, 0x0, 0xbf2f288e00000000, 0x517eb7b3726, 0x32c7920)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:237 +0x7d
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/golang.org/x/net/http2.(*Framer).ReadFrame(0xc420666000, 0xc422ea1e20, 0xc422ea1e20, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/golang.org/x/net/http2/frame.go:492 +0xa6
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*http2Client).reader(0xc420668000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:1123 +0x117
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:265 +0xb41
May 15 08:11:48 <hostname> dockerd[765]: goroutine 68 [select]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*controlBuffer).get(0xc420650040, 0x1, 0x0, 0x0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/controlbuf.go:289 +0x135
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc/transport.(*loopyWriter).run(0xc420858600)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/controlbuf.go:374 +0x1be
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client.func3(0xc420668000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:298 +0x7e
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc/transport.newHTTP2Client
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/transport/http2_client.go:296 +0xc91
May 15 08:11:48 <hostname> dockerd[765]: goroutine 82 [select, 93 minutes, locked to thread]:
May 15 08:11:48 <hostname> dockerd[765]: runtime.gopark(0x2674cd0, 0x0, 0x1a6b2b3, 0x6, 0x18, 0x1)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/proc.go:291 +0x120
May 15 08:11:48 <hostname> dockerd[765]: runtime.selectgo(0xc4207b4f50, 0xc4206a7860)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/select.go:392 +0xe56
May 15 08:11:48 <hostname> dockerd[765]: runtime.ensureSigM.func1()
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/signal_unix.go:549 +0x1f6
May 15 08:11:48 <hostname> dockerd[765]: runtime.goexit()
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/asm_amd64.s:2361 +0x1
May 15 08:11:48 <hostname> dockerd[765]: goroutine 69 [chan receive, 10 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/pkg/signal.Trap.func1(0xc4201e20c0, 0x2683a20, 0xc42003a230, 0xc4206d7b80)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/pkg/signal/trap.go:38 +0x5d
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/pkg/signal.Trap
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/pkg/signal/trap.go:36 +0x120
May 15 08:11:48 <hostname> dockerd[765]: goroutine 43 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*ccBalancerWrapper).watcher(0xc420176580)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:122 +0x14c
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.newCCBalancerWrapper
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/balancer_conn_wrappers.go:113 +0x151
May 15 08:11:48 <hostname> dockerd[765]: goroutine 71 [chan receive, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap.func1(0xc42064e540, 0x1a7a122, 0xf)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:18 +0x46
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/daemon.(*Daemon).setupDumpStackTrap
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/daemon/debugtrap_unix.go:17 +0xc1
May 15 08:11:48 <hostname> dockerd[765]: goroutine 76 [IO wait, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.runtime_pollWait(0x7f1189898c90, 0x72, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/runtime/netpoll.go:173 +0x59
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*pollDesc).wait(0xc42065e898, 0x72, 0xc420719a00, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x9d
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*pollDesc).waitRead(0xc42065e898, 0xffffffffffffff00, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x3f
May 15 08:11:48 <hostname> dockerd[765]: internal/poll.(*FD).Accept(0xc42065e880, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/internal/poll/fd_unix.go:372 +0x1aa
May 15 08:11:48 <hostname> dockerd[765]: net.(*netFD).accept(0xc42065e880, 0xc42006de58, 0x411dda, 0x30)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/fd_unix.go:238 +0x44
May 15 08:11:48 <hostname> dockerd[765]: net.(*UnixListener).accept(0xc42089e270, 0x53810c, 0x24eee60, 0xc4206fbe60)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/unixsock_posix.go:162 +0x34
May 15 08:11:48 <hostname> dockerd[765]: net.(*UnixListener).Accept(0xc42089e270, 0xc420042028, 0x22fb680, 0x32490f0, 0x2633d20)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/unixsock.go:253 +0x4b
May 15 08:11:48 <hostname> dockerd[765]: net/http.(*Server).Serve(0xc4206d8d00, 0x26a4480, 0xc42089e270, 0x0, 0x0)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/http/server.go:2773 +0x1a7
May 15 08:11:48 <hostname> dockerd[765]: net/http.Serve(0x26a4480, 0xc42089e270, 0x2684600, 0xc42089e2a0, 0x43465a, 0x2674b60)
May 15 08:11:48 <hostname> dockerd[765]:         /usr/local/go/src/net/http/server.go:2392 +0x75
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/daemon.(*Daemon).listenMetricsSock.func1(0x26a4480, 0xc42089e270, 0xc42089e2a0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/daemon/metrics_unix.go:31 +0x4d
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/daemon.(*Daemon).listenMetricsSock
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/daemon/metrics_unix.go:30 +0x195
May 15 08:11:48 <hostname> dockerd[765]: goroutine 77 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).watcher(0xc42089eab0)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:109 +0x184
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*ccResolverWrapper).start
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/resolver_conn_wrapper.go:95 +0x41
May 15 08:11:48 <hostname> dockerd[765]: goroutine 44 [select, 93 minutes]:
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).transportMonitor(0xc4208f2000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:1373 +0x23d
May 15 08:11:48 <hostname> dockerd[765]: github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect.func1(0xc4208f2000)
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:949 +0x1b7
May 15 08:11:48 <hostname> dockerd[765]: created by github.com/docker/docker/vendor/google.golang.org/grpc.(*addrConn).connect
May 15 08:11:48 <hostname> dockerd[765]:         /go/src/github.com/docker/docker/vendor/google.golang.org/grpc/clientconn.go:940 +0xe3
May 15 08:11:48 <hostname> dockerd[765]: goroutine 46 [IO wait, 4 minutes]:

Has anybody experienced similar issues that cause Docker swarm to behave this way?