Hello,
This is my first time posting here.
I recently began experimenting with docker swarm as a migration path away from kubernetes, and am trying to achieve some of the same high-availability for apps and databases which I had with kubernetes.
As such, I’m trying to deploy etcd as a distributed KV-store, for use with high-availability database solutions (Patroni, ++).
I’d like to have a virtual IP (VIP) or url where clients can reach a random etcd-replica, and also some templatable (or similar) way to express the IPs or urls of all replicas, within each replica. Basically so each etcd replica knows how to reach every other replica (without the randomness of using the shared service VIP). The etcd image only has etcd commands available, no curl/nslookup/dig.
What should be done?
- Run HAproxy in the same namespace, with
tasks.serviceNameso the HAproxy can know the IPs of upstreametcdreplicas. (This will introduce an extra hop for comms between etcd replicas, I think). - Run busybox or something else as an init container, using
nslookupor similar to populate an/etc/hostsfile mounted as a shared volume between the init container andetcdreplicas. (This feels somewhat brittle. What happens when one replica restarts and gets a new IP + url?) - Something else (please advise)
Here’s my current best attempt, but every node fails to reach eachother:
---
services:
app:
image: quay.io/coreos/etcd:v3.7.0
hostname: '{{.Service.Name}}.{{.Task.Slot}}'
# depends_on:
# - init-volume-permission:
# "condition: service_completed_successfully"
command: |
etcd
environment:
- ETCD_NAME=etcd.{{.Task.Slot}}
- NAME=${ETCD_NAME}
- ETCD_LISTEN_PEER_URLS=http://0.0.0.0:2380
- ETCD_LISTEN_CLIENT_URLS=http://0.0.0.0:2379
- ETCD_INITIAL_ADVERTISE_PEER_URLS=http://{{.Service.Name}}.{{.Task.Slot}}:2380
# - ETCD_INITIAL_CLUSTER=etcd.1=http://{{.Service.Name}}:2380,etcd.2=http://{{.Service.Name}}:2380,etcd.3=http://{{.Service.Name}}:2380
- ETCD_INITIAL_CLUSTER=etcd.1=http://{{.Service.Name}}.1:2380,etcd.2=http://{{.Service.Name}}.2:2380,etcd.3=http://{{.Service.Name}}.3:2380
- ETCD_INITIAL_CLUSTER_STATE=new
- ETCD_INITIAL_CLUSTER_TOKEN={{.Service.Name}}
- ETCD_ADVERTISE_CLIENT_URLS=http://{{.Service.Name}}:2379
- ETCD_DATA_DIR=/etcd-data
networks:
internal:
# ports:
# - "2379:2379"
# - "2380:2380"
deploy:
replicas: 3
labels:
- "traefik.enable=false"
restart_policy:
condition: on-failure
placement:
max_replicas_per_node: 1
constraints:
- node.role == manager
- node.labels.etcd == true
preferences:
- spread: node.labels.etcd
# endpoint_mode: dnsrr
update_config:
parallelism: 3
healthcheck:
test: ["CMD", "etcdctl", "endpoint", "health", "http://localhost:2379"]
interval: 30s
timeout: 10s
retries: 10
start_period: 300s
init-volume-permission:
image: busybox:latest
command: sh -c "chmod -R 0700 /etcd-data"
volumes:
- etcd:/etcd-data
deploy:
replicas: 3
labels:
- "traefik.enable=false"
restart_policy:
condition: on-failure
max_attempts: 2
placement:
max_replicas_per_node: 1
constraints:
- node.role == manager
- node.labels.etcd == true
preferences:
- spread: node.labels.etcd
init-replicas-hosts:
image: busybox:latest
command: sleep 500
hostname: "{{.Service.Name}}.{{.Task.Slot}}"
networks:
internal:
deploy:
replicas: 3
labels:
- "traefik.enable=false"
placement:
max_replicas_per_node: 1
constraints:
- node.role == manager
- node.labels.etcd == true
preferences:
- spread: node.labels.etcd
networks:
internal:
attachable: true
# driver_opts:
# dsr: "true"
volumes:
etcd: