Not sure if there is a feature for this, hence my question:
I use Docker Compose to build a dev environment. I work with big data tools that need to scale, like Zookeeper, Kafka, stream processing engine etc…
Compose is great to spin a cluster quickly and run it on one machine; links allow to link containers so they can talk to each other, but outside the container, it’s opaque.
Of courseI I can expose the ports to the outside, but I can’t ‘fix’ a port on a scalable container: it is assigned automatically.
That becomes a pain when running with external tools: either I bind a random port to one IP and I need to figure out ALL the ports assigned every time I rebuild the environment. Or i can assign an IP and port, but then I can’t auto-scale anymore.
Ideally I’d like to ‘scale’ container with a naming scheme I define, so that external IP is assigned automatically and all I need is know the name. For example if I have 3 Zookeeper nodes, and the scheme is zk#, then I know they can be reached at their standard port on zk1, zk2 and zk3 for example… generating etc/hosts file and such
That makes life a lot easier.
Is there a way to do such a thing easily? Or does it just need to be scripted?
I can script the generation of the docker-compose.yml to do that, but that kinda defeats the purpose of the ‘scale’ function.
There has not been any good way to do this in Docker… at least not that I know of.
Zookeeper 3.5 now supports cluster re-assignment so it’s possible to run the command once the containers are up, but it’s not ideal.
In the end, for ZK at least, a fi set of 3 or 5 nodes is fine. For Kafka, I use ZK to register the nodes, and I had created ID by using the container IP without the dots. This works well to get unique IDs, but causes other problems since IP changes on restart… I kinda gave up on that.
I use Kubernetes now; it doesn’t solve everything but definitely helps as Services map to the pods/nodes of each container, and can be probed from within the container with the API.
You may take a look of FireCamp. It supports Zookeeper, Kafka, etc. It assigns a unique name for each zk node. For example, the swarm cluster name is c1, zk service name is myzk. FireCamp assigns a unique name for each zk node, myzk-0.c1-firecamp.com, myzk-1.c1-firecamp.com, myzk-2.c1-firecamp.com. The zk container listens on the standard zk port.
While, FireCamp only works on AWS currently. The cluster is deployed via AWS CloudFormation, not Docker compose yet.