I’m a relative newbie when it comes to docker swarm mode, and I’d appreciate it if anyone can point out my mistake.
I’m trying to get open source docker v17.06 to launch my 3 containers on 3 specific nodes in the swarm using docker stack deploy and placement constraints, but find my constraints are always ignored, and docker tries to bring all 3 containers up on the same node, the management node.
Here is my docker-compose.yml:
I’ve tried using node.hostname, node.id, engine labels, nothing seems to get the 2nd and 3rd containers to start on their assigned nodes. I can’t find any examples on the net that actually work for me.
Here is my docker info:
I’m not sure about the requirements to run zookeeper, but as far as docker composition is concerned mode: global and placement: do not go together under a service config. Either you use mode: global - or - mode: replicated + placement: (optional).
When you specify 3 services (dev2-1, dev2-2, dev3-3) in mode: global, you’re telling docker to run every dev2-? in ALL existing nodes. That means, a total of 9 (3 x 3 dev2-?).
Try to replace mode: global with mode: replicated and replicas: 1. You may choose not specify any and the default is replicated and replicas = 1.
Also, you may want to specify hostname: in the service config to correspond to - node.hostname == mynode1.
I tried the changes you suggested - added hostname, changed mode to replicated, added replicas. It has no effect. The dev2-2 container is trying to come up on the mgmt node, mynode1, and fails because the dev2-1 container is already using port 22281 there.
$ docker stack deploy -c docker-compose-dev2.yml zookeeper
Creating network zookeeper_default
Creating service zookeeper_dev2-1
Creating service zookeeper_dev2-2
Error response from daemon: rpc error: code = 3 desc = port '22281' is already in use by service 'zookeeper_dev2-1' (fafortsdnuew826q6eijctqqm)
You cannot have conflicting host ports. The config says that all 3 instances are asking the host to bind to the same ports. Try to specify different port numbers for all instances. Or perhaps you might not want to know which host port to bind to and let the host bind to arbitrary port numbers. So you want to drop the host part and specify as follows for all the instances:
The issue isn’t zookeeper. The issue is that my constraints are being ignored, and dev2-2 should be launched on mynode2, where it won’t have a port conflict with dev2-1 or dev2-3.
The cluster works fine when I launch via docker run on each host, I just haven’t figured out what magic is required to get docker stack deploy to launch my containers on those 3 hosts.
I saw a few zookeeper examples on github with a docker-compose.yml file, but docker-compose only run containers on one host, it tells me to use docker swarm deploy to launch on multiple machines. I wondered why I hadn’t found any examples for the latter. Now I know why.
Thanks, but that person had a different issue, one where the local host wasn’t using 0.0.0.0 to refer to itself in the zookeeper config file. I have that issue already solved in my entrypoint script.
I am trying now to use his examples of 3 docker service create commands, but I am still running into the same issue, the second service refuses to start, complaining a port is already in use.
This one port per service per swarm seems like an odd restriction. So if I have a cluster of 1000 nodes, I can only run one httpd service listening on port 80 in the entire swarm, I can’t have multiple services using different IP addresses?
I guess it’s time I stopped using google and started searching for a good book on docker best practices, if there is one.
Actually, he isn’t exposing ports at all, which lets the containers come up, and talk to each other, but is not helpful because no other program (outside another docker service perhaps) can communicate with the zookeeper application. If everything in the world ran inside docker already, that would be great, but it’s a chicken and egg situation.
They’re conflicting because each one of these service definitions tells Swarm “open this port on each host, to be routed in to the specified port in this service’s containers”. L4 LB (using IPVS) then happens automatically across replicas when ingress occurs on the exposed port.
Most likely you want to put them all on the same docker network and do ZooKeeper-to-ZooKeeper communication using that network and swarm mode’s built-in service discovery. Then the services which you don’t want to expose to the outside world will not have port conflicts, they will communicate with each other using Docker’s private overlay networking.
If you have other services that need to access the ZooKeeper instances they will go on this docker network as well.
It’d be preferable as well if you could roll the non-leader ZooKeeper instances into just one service using replicas: 2 (as well as having a leader service on the same network). But, I am aware of some funny issues related to hostname / bind address that do arise in such a model when using some tech, so I can see how that might not work.
I am hitting exactly the same problem. In my case I want to deploy 3 web services which form a distributed app. Each needs to listen on port 443 and although they are being constrained to different hosts, the docker stack deploy command fails to start with:
Error response from daemon: rpc error: code = 3 desc = port '443' is already in use by service 'votingapp_spdz-proxy2' (zl2nqx7ky93kxqsa8athtihnz)
I’m facing the same problem and found that this deploy options is ignored by docker-compose, it says that deploy configuration only takes effect when using docker stack deploy, and is ignored by docker-compose.