Single task services

General Discussions Feature Requests
1

I see I cannot use “docker run” on a swarm overlay network. As a workaround, I can use “docker service create --replicas=1 ...” to run a single task service. As I understand it, the service has a VIP and the task has its own IP on the overlay network. Also, the service can be scaled to more replicas later even if the image doesn’t support scaling such as a mysql container which uses a host mount for the database (only one mysql container can manage the database).

I’d like to see a new feature to define single task services where the service uses the task’s IP as the VIP and the service cannot be scaled (attempt to later scale generates an error).

Perhaps use syntax: “docker service create --mode=single ...”

2

What’s the use case in mind here?

3

Basically, many of my services in my deployment are single task. The proxies are deployed in pairs with one proxy being primary and the other proxy for backup. I run these proxies as single task services with a constraint specifying different nodes in the cluster. I have a proxy pair sitting in front of nginx/php service (which has multiple replicas). The php service has a local haproxy to connect to a pair of database proxies (primary/backup) for mysql and redis databases. The pair of database proxies are also single task as I want to have all connections go through a single proxy (in the case of mysql, the proxy is implemented with ProxySQL and for redis, the proxy is HAProxy).

There are 5 clusters of 2 MySQL servers sitting behind each ProxySQL proxy. One of the MySQL servers in a cluster is the Master, while the other is a replication Slave. I run these database clusters with 10 single task services with a constraint specifying the node where the database data resides on the host using very fast NVMe SSDs on the host.

A final MySQL server is deployed to consolidate the 5 clusters’ databases into one instance using MySQL 5.7 multi-source replication. In the early morning, admin tasks update this consolidated database using the changes made during the day and produce new content for the next day. At 5am, the website is marked as OFFLINE for maintenance and the 5 clusters are taken offline to resync the databases with the consolidated database and the clusters are restarted with the updated databases. The 5 clusters are not resync’d all at the same time as 1 cluster is taken down, resync’d, and then restarted in a rolling update so only 1 cluster is offline at a time.

I also have similar proxies for IMAP and LMTP with the proxy routing connections to the appropriate Dovecot backend servers. Each backend Dovecot server is deployed as a pair of single task services (one for primary user access and the second one for backup). There are many pairs of Dovecot servers that will be deployed. Each pair is also replicated to a single consolidated Dovecot multi-source server that is used to perform mailbox maintenance during the night and resync’d back to the pairs of live Dovecot servers.

Anyway, the entire swarm is fairly large but primarily consists of pairs of single task services (some managing traffic through the system and some managing persistent data). My swarm is currently in development and hasn’t been put into production yet.

As for my feature request, I primarily want the “docker service create ...” command to deploy these pairs of single task services where there is no possibility that someone in production will attempt to scale one of these services by mistake.

As “docker service create...” already supports the --mode option to select whether the service is replicated or global, I figure this option should be extended to support more modes. In this case, I’m suggesting single as a mode choice.

I would also like to see a mode that would allow me to specify the service is deployed as a pair of tasks with the first task being primary and the second task being backup, but having DNS requests return the primary task’s IP if it is online or return the second task’s IP. This would allow me to probably deploy my proxy/database pairs as a single service rather than two single task services and have to use a local proxy for failover. If Docker were to add such a mode, the service should support 2 or more replicas so that the task IP selected cascades to the next replica if all the first replicas are OFFLINE.

4

Interesting. Sounds like it might be worth opening a ticket on https://github.com/docker/docker or https://github.com/docker/swarmkit to describe your use case. I’m not sure if --mode single is the exact solution the Swarm maintainers would recommend or endorse, but it would probably help them to plan for the future if they knew the requirements in your use case. Sounds like you have a quite particular way that you’d like things laid out and they might be able to help work towards the right primitives to get there with you.

5

The other way to support --mode single services is to simply support “docker run” on swarm overlay networks. These classic containers would then become single task services in the swarm. That would take care of my original feature request.

Also, I’m thinking what is also really needed is to support more of the HAProxy load balancing capabilities for services.

For instance, add support for --backup-replicas=n to specify the number of replicas that are in BACKUP to the other primary replicas. If all the primary replicas are down, the DNS query would return the IP of the first available backup replica. HAProxy also gives the ability to specify the RISE/FALL/INTERVAL for marking a task as available or offline for load balancing. It would be good if Docker Services had the same BACKUP and RISE/FALL semantics, only implemented via the internal DNS for the service (and to define RISE/FALL/INTERVAL differently for each service created).

Another nice feature would be to have the HealthCheck be able to return an integer to the internal DNS that could be used for selecting the IP of the task that has the lowest value of the most recent HealthChecks. Then, Docker Services could support a mode for least-loaded task. The HealthCheck script would determine how much load the container is currently processing and return an integer that reflects this current load based on some numeric scale of its choosing (e.g., the integer returned could be the number of connections the container is currently servicing).

Or, the integer returned could be interpreted as a Priority value where the highest priority task (lowest integer value) would have it’s IP returned from the internal DNS. I could then have my Primary task return zero while the Backup task script returns 1.

If Docker Services supported the load balancing that many reverse proxies support (only implemented through the internal DNS) I could deploy and scale my website using many fewer service definitions and fewer reverse proxies to control service connections.