I observed the following behavior:
I ran docker swarm and a docker registry on one VM. When running docker push to publish a big docker image to the docker registry, sometimes I found all docker swarm service restarted, and I saw the following errors in docker log:
*level=error msg=“heartbeat to manager {**} failed” error=“rpc error: code = InvalidArgument desc = session invalid”**method="(session).heartbeat" module=node/agent
After some research looks like the dispatcher-heartbeat parameter of docker swarm is related, and the issue is gone after I increase it from the default 5 sec to 10 sec.
I suspect during the docker push of the big docker image, the network is very busy so the swarm heartbeat packet is lost or delayed. And even though there is only one node in the swarm cluster, there is still heartbeat (a loopback from the dockerd to itself?).
Can somebody help to confirm my understanding?
Thanks.