Docker, Compose, External IP and Scaling

Not sure if there is a feature for this, hence my question:

I use Docker Compose to build a dev environment. I work with big data tools that need to scale, like Zookeeper, Kafka, stream processing engine etc…

Compose is great to spin a cluster quickly and run it on one machine; links allow to link containers so they can talk to each other, but outside the container, it’s opaque.

Of courseI I can expose the ports to the outside, but I can’t ‘fix’ a port on a scalable container: it is assigned automatically.

That becomes a pain when running with external tools: either I bind a random port to one IP and I need to figure out ALL the ports assigned every time I rebuild the environment. Or i can assign an IP and port, but then I can’t auto-scale anymore.

Ideally I’d like to ‘scale’ container with a naming scheme I define, so that external IP is assigned automatically and all I need is know the name. For example if I have 3 Zookeeper nodes, and the scheme is zk#, then I know they can be reached at their standard port on zk1, zk2 and zk3 for example… generating etc/hosts file and such

That makes life a lot easier.

Is there a way to do such a thing easily? Or does it just need to be scripted?
I can script the generation of the docker-compose.yml to do that, but that kinda defeats the purpose of the ‘scale’ function.

I apreciate your thougths on this.


1 Like

Any news about this feature request?

There has not been any good way to do this in Docker… at least not that I know of.

Zookeeper 3.5 now supports cluster re-assignment so it’s possible to run the command once the containers are up, but it’s not ideal.
In the end, for ZK at least, a fi set of 3 or 5 nodes is fine. For Kafka, I use ZK to register the nodes, and I had created ID by using the container IP without the dots. This works well to get unique IDs, but causes other problems since IP changes on restart… I kinda gave up on that.

I use Kubernetes now; it doesn’t solve everything but definitely helps as Services map to the pods/nodes of each container, and can be probed from within the container with the API.


You may take a look of FireCamp. It supports Zookeeper, Kafka, etc. It assigns a unique name for each zk node. For example, the swarm cluster name is c1, zk service name is myzk. FireCamp assigns a unique name for each zk node,,, The zk container listens on the standard zk port.

While, FireCamp only works on AWS currently. The cluster is deployed via AWS CloudFormation, not Docker compose yet.