Docker swarm to manage python microservices "farm"

General
1

Hi !

I read since days without finding what I want ..

I built python microservices “farm” where I can deploy autonomous microservice in container with shell script including input specific parameters

I build specific image for each deployed microservice with dockerfile..

  • each service has one dockerfile in a dedicated directory and one dedicated image
  • each container is launched depending of the input stream
  • each service listen to a webhook
  • each webhook is served by nginx reverse proxy mode .

I can deploy hundred of containers with my project .. number of running services can move across time (new vs stopped one)

but I struggling to see if swarm can help in this architecture. It would be nice to benefits of load balancing system btw..

I search and read a lot of docker-compose.yaml examples without founding one that meet my requirements ..

any help appreciated :slight_smile:

2

Swarm can help if you want to run the services on multiple hosts.

Swarm itself does not have any auto-scaling features. Creating more services on demand is not handled by Swarm, for that you need additional tooling.

3

thank you for your reply :slight_smile:

5

Hello,

Use a common base image for services (configured via ENV).

Deploy services via docker stack deploy or Docker SDK.

Use Traefik for dynamic routing and service discovery.

Swarm works best with replicated services, not lots of unique, one-off containers. Consider adapting your setup for that. Want a quick example docker-compose.yml for Swarm + Traefik?

Best Regard,
lexev

6

It is unclear what swarms role is supposed to be in your architecture.

Where swarm can help:

  • It could help by placing services on different nodes
    • this requires the images to be hosted on a container image registry
    • Since a container can be scheduled on any node, local volume data is not going to work, if you need a container needs to puck up its previous state from a volume. This would work with a remote nfs or cifs share though.
  • it supports to run identical replicas of a service, that are internally exposed through a service vip and balanced in a round-robin manner by the kernel loadbalancer IPVS. If you publish a port on the service, it will point to the service vip.

You need to share more (and less ambiguous) information of what the solution should be able to do.

7

Hi !

thanks for your reply :slight_smile:

I don’t use replicated services since each of my services are unique as @tim753milner says “Swarm works best with replicated services, not lots of unique, one-off containers.”

I will take a look on Traefik, thank you :wink:

@meyay I was unclear about how to use swarm but now it’s ok , I see I can’t use it in my project

thanks !