Docker Community Forums

Share and learn in the Docker community.

Large Docker Swarm fails to schedule all containers


(Deepeddy) #1

I am attempting to schedule several hundred containers on a Docker Swarm. Docker version 17.06.2-ce client and server. Docker is in “Swarm” mode. Docker-Compose version 1.16.0 builds deploys containers specified in the YAML compose file, version 3.3.

I use the docker deploy stack abstraction:

stack deploy --compose-file docker-compose.yml myApp

I have a Swarm cluster composed of many VMs in a cloud. When the application is deployed all containers are listed as “running” but some never get “scheduled” to a swarm node. Here a snippet of the listing output.

m92w3lx2llfs query_C4-78.1 deepeddy/mysqlite:worker
Running New 3 minutes ago

o2mvzz60etf0 query_C4.1 deepeddy/mysqlite:worker swarmmaster Running Running 3 minutes ago

NOTICE query_C4.1 as been scheduled on node swarmmaster. query_C4-78.1 is running but never scheduled.

I also observe that the same number of containers are scheduled out of 200 started every time, around 124 containers out of 196. After 124 containers get scheduled to nodes the remaining containers are not scheduled, although the “docker service ps” reports they are “running”. Other listing seem to indicate these unscheduled containers have been initialized but have not progressed to the next state: pending (waiting to be scheduled). This seems to suggest their is a RESOURCE threshold of some type that
prevents Swarm from scheduling further containers. There appears to be no failures or logs reporting failures.

Does Docker Swarm have a config file that specifies resource thresholds? I have tried adding additional nodes to the swarm with no success. The cutoff is always 124 containers scheduled.