Docker creating other dockers


I would like to build an architecture in which there is main docker that runs all the time and fetch data from database. As soon as it finds old entries, it spawns number of other dockers and provide to each of them list of old entries so that these dockers could run in parallel and execute all required job. E.g. if main docker app detect 100 entries that needs to be updated, it is going to spawn 10 child-containers to concurrently update all the entries in db.

Can you please advise if such architecture is feasible to be implemented in docker / do you have any examples / documentation that you could share.

Thanks, M.


This sort of thing is very feasible. This sounds like a simple worker queue setup. You could use docker-compose to designate the types of services you want, and then use the docker-compose scale feature to change the number of workers. You would have to provide the queueing mechanism (rabbitmq, redis, beanstalk), the master process to add data and scale workers, and the workers themselves.

Look into existing queueing systems and see if any hits pop up when you search for docker things specific to each system type.