Simultaneous increase and drop of container memory usage in Docker Swarm?

Hi there,

we are using a docker swarm environment with 11 Nodes. ATM this swarm is running about 350 services with round about 500 containers. Everything works smooth. But yesterday something strange happened and the memory usage of ALL containers on ALL nodes increased simultaneously and also decreased simultaneously a few minutes later (see screenshot from grafana/prometheus chart below). The infrastructure monitoring shows no evidence of anything abnormal. What can be the reason for that behaviour ?

Thanks a lot in advance.
René

Hi reneboehm.
I want to ask you. If you know please help me
Current we’re running docker swarm with 4 node :

  • 3 node master + 1 node worker.
  • All 4 node docker server running VM with mode global
  • Os : Ubuntu 16.04.5 LTS
  • Docker version : 18.09.0
  • We use gitlab to run CI/CD deploy service via docker stack with healthcheck
    Occasionally we encounter problems like this:
  • When one node in docker swarm load very high and can’t resolve and other node cpu usage very low.
  • We tried kill container and when container healthy but it continue to load high and experience endless looping.
    Have you ever encountered this problem

You hijacked my thread :wink:

Anyway…does the master node have a high load at the time ? Is the high load an iowait issue ? When you restart the container, does the swarm schedule on another node with lower load ?