we are using a docker swarm environment with 11 Nodes. ATM this swarm is running about 350 services with round about 500 containers. Everything works smooth. But yesterday something strange happened and the memory usage of ALL containers on ALL nodes increased simultaneously and also decreased simultaneously a few minutes later (see screenshot from grafana/prometheus chart below). The infrastructure monitoring shows no evidence of anything abnormal. What can be the reason for that behaviour ?
Anyway…does the master node have a high load at the time ? Is the high load an iowait issue ? When you restart the container, does the swarm schedule on another node with lower load ?