Docker-compose scale in swarm scheduling always same node

Docker Engine Compose
1

Hi, I have a swarm with one master and three agents. When issuing a docker-compose up command followed by a docker-compose scale www=n (I have a www, redis and loadbalance container) the new www containers will always be created on the same agent. It goes to the point where there are no more resources available to schedule container (even with 2 agent nodes completely free). I was under the impression the new container should be spread along the free-er nodes according to my swarm strategy (which is spread).
I’m on Docker Machine with Windows 10 host. Any ideas?
Thank you!

2

Got a similar issue here, did you figure it out?
thanks!

3

Had the same issue.
In my case the ‘problem’ was that the necessary image was being built by compose on the first run. That was done towards a specific selected node. When you scale up afterwards that 's the only node on which the image is available.
It should work better when images are pulled from a repository and not built by compose (which is a dev thing anyway).
Workaround for me:

  • create a affinity in the compose file towards another node
  • perform the docker-compose build service
  • remove the affinity
  • docker-compose scale service=x will now use the other node

I saw that UCP makes sure that necessary images are available on other nodes by performing some magic

4

I was told a workaround is to use multi-host networking, didn’t try it yet. Just FYI here.

5

Hi,

mult-host networking and creating an overlay network (from the compose file) are a requirement if you want the defined services to be able to communicate with each other (the usage of --link on docker run or links in compose files). This is now deprecated (see also https://docs.docker.com/engine/userguide/networking/default_network/dockerlinks/).

Docker-compose by default (version 1.6) creates a network for an application stack and when it determines that multi-host networking is configured it should create an overlay network.

Since docker-compose is just working on top of standard docker api and docker swarm does the same, I think it’s kind of logical what happens.

  • docker-compose performs the build of the image and asks the api through the swarm to do so
  • the swarm schedules the build on the next available docker engine and the build-context is transferred
  • this docker engine now has the necessary image available
  • docker-compose wants to run this image and asks swarm to do so
  • either swarm or docker-compose knows of the dependency on the image and thus the docker run will only work against that specific engine?

I have no idea where this knowledge about the runtime is registered. Docker-compose has some kind of memory e.g. it knows how many instances to start when you ask for a scale service=n

Hope I have helped readers with this explanation and hope It makes sense :slight_smile:
It helped myself in thinking about it :wink:

regards,
Rick

1 Like
6

Agree. Docker swarm is under layer component and I also think Docker-Compose should just manage it to create whatever network needed to get the service working. :slight_smile:
I actually reported a scale related issue to UCP team and was told that they will fix it.
FYI & thanks.

7

Hi,

I already experienced that UCP works some magic. It does scale (when you use docker-compose CLI). Behind the scenes it seems to make sure that the image will be built on all necessary nodes or it copies them in some other way (save/load behind the scenes?)

regards,
Rick

8

Yes I saw that too, docker-compose CLI seems scale on the available nodes automatically, it may just do docker run on each nodes then it will pull necessary images I guess. Originally I saw all new containers are running on same host is actually because my cloned VM have same docker ID, that confused UCP that’s why it all goes to master node only.
BTW the specific issue I was seeing is the “affinity” parameter in docker-compose is not working as expected in UCP environment. FYI

9

We had this problem where all instances ended up on the same node. We’re using a docker-compose stack with

docker stack deploy -c ourstack.yml prod

After doing a deploy with

docker service update --image prod_theimage

We would find all the instances on the same node. The solution was to add rolling restarts.

docker service update --update-parallelism 1 --update-delay 10s --image prod_theimage