Docker Community Forums

Share and learn in the Docker community.

Can UCP use Consul as K/V store?


(Jojojojo1234) #1

Hi,

Was bit surprised when I installed UCP and default K/V backend was etcd. The thing is i manually (w/o docker-machine) installed my Swarm cluster with Consul as my K/V store.

I went further and installed UCP. UCP installer created its own “ucp-swarm-join” containers on each swarm node, so literally created its own swarm cluster with etcd store.

Obviously now i have 2 virtual swarm clusters, one my own running Consul and second UCP with etcd.

Is there option to install UCP and let it use my pre-defined K/V, in this case Consul?

Thanks a lot


(Vivek Saraswat) #2

At this time, UCP beta uses etcd for our HA setup, so we don’t have an option for using other pre-defined K/V stores. The goal is to streamline deployment, maintainability, and management. Longer term are there particular use cases you are looking for for having custom k/v backends?


(Jojojojo1234) #3

Thanks for your quick response. I got confused when i saw etcd installed and thought i can’t use any other KV besides that. Which seem not to be true. If UCP uses etcd internally for its HA that is ok to me.
We decided to use Consul as our KV store for services endpoints and variables. I will just hook up Consul to swarm for our use case regardles of what UCP uses as KV backend.

Cheers!


(Vivek Saraswat) #4

Ah no problem, yes etcd is just internally used for UCP and shouldn’t conflict with K/V’s you want to use for other purposes. If you do run into any problems please let us know!


(Jojojojo1234) #5

Hi Vivek,

When i install UCP controller on Swarm manager node, it installs etcd K/V as well on that particular node. Then i added second replica to UCP to achieve HA. Problem is when first Swarm manager node goes down, its etcd K/V goes down too. In other words second HA replica node won’t become master because it can’t reach etcd K/V.

K/V store definitelly has to be on its own individual node (best HA cluster). My question now is: I would like to use our existing Consul K/V store. I can’t find anywhere how to install UCP with custom K/V store except of default etcd. Could you help here please?

Ta


(Vivek Saraswat) #6

Hi Jojo, you cannot use UCP beta with a custom k/v store as of now. In fact we are treating etcd as a backend service that you should not need to access except under specific support circumstances (see http://ucp-beta-docs.s3-website-us-west-1.amazonaws.com/kv_store/).

However, it’s odd that the HA replica node isn’t becoming master, as each replica should contain a copy of etcd k/v as well. One question, are you running a master + 2 replicas? You need a minimum of 3 nodes to avoid split-brain HA decision-making issues.


(Jojojojo1234) #7

Hi. I only created 1 replica. I read later yesterday that 3 master nodes are required to avoid split-brain just like you said. I will try today. Thanks for the hint!


(Jojojojo1234) #8

Hi. Got stuck at one thing when having multiple replicas. When you configure docker engine to use overlay network, you should pass this into docker sysconfig:

OPTIONS="–cluster-advertise “NODE_IP”:12376 --cluster-store etcd://“ETCD_IP”:12379 --cluster-store-opt kv.cacertfile=/var/lib/docker/discovery_certs/ca.pem --cluster-store-opt kv.certfile=/var/lib/docker/discovery_certs/cert.pem --cluster-store-opt kv.keyfile=/var/lib/docker/discovery_certs/key.pem"

Since now i have multiple etcd, what IP should i use for “ETCD_IP” ? There is no such thing like global cluster IP, only individual IPs of master and master replicas that hold K/V.

Any ideas?

Thanks again