Docker, Compose, External IP and Scaling

Not sure if there is a feature for this, hence my question:

I use Docker Compose to build a dev environment. I work with big data tools that need to scale, like Zookeeper, Kafka, stream processing engine etc…

Compose is great to spin a cluster quickly and run it on one machine; links allow to link containers so they can talk to each other, but outside the container, it’s opaque.

Of courseI I can expose the ports to the outside, but I can’t ‘fix’ a port on a scalable container: it is assigned automatically.

That becomes a pain when running with external tools: either I bind a random port to one IP and I need to figure out ALL the ports assigned every time I rebuild the environment. Or i can assign an IP and port, but then I can’t auto-scale anymore.

Ideally I’d like to ‘scale’ container with a naming scheme I define, so that external IP is assigned automatically and all I need is know the name. For example if I have 3 Zookeeper nodes, and the scheme is zk#, then I know they can be reached at their standard port on zk1, zk2 and zk3 for example… generating etc/hosts file and such

That makes life a lot easier.

Is there a way to do such a thing easily? Or does it just need to be scripted?
I can script the generation of the docker-compose.yml to do that, but that kinda defeats the purpose of the ‘scale’ function.

I apreciate your thougths on this.

Thanks

1 Like

Any news about this feature request?

There has not been any good way to do this in Docker… at least not that I know of.

Zookeeper 3.5 now supports cluster re-assignment so it’s possible to run the command once the containers are up, but it’s not ideal.
In the end, for ZK at least, a fi set of 3 or 5 nodes is fine. For Kafka, I use ZK to register the nodes, and I had created ID by using the container IP without the dots. This works well to get unique IDs, but causes other problems since IP changes on restart… I kinda gave up on that.

I use Kubernetes now; it doesn’t solve everything but definitely helps as Services map to the pods/nodes of each container, and can be probed from within the container with the API.

2 Likes

You may take a look of FireCamp. It supports Zookeeper, Kafka, etc. It assigns a unique name for each zk node. For example, the swarm cluster name is c1, zk service name is myzk. FireCamp assigns a unique name for each zk node, myzk-0.c1-firecamp.com, myzk-1.c1-firecamp.com, myzk-2.c1-firecamp.com. The zk container listens on the standard zk port.

While, FireCamp only works on AWS currently. The cluster is deployed via AWS CloudFormation, not Docker compose yet.