Docker Community Forums

Share and learn in the Docker community.

Trouble setting up loadbalancer with UCP

ucp

(Hmaeck) #1

I’m having trouble configuring the proxy on my test UCP setup.
I’ve got a 3 node test setup and only a single controller (so no HA)

I downloaded the UCP client bundle inside my controller node (192.168.123.14, boot2docker) and put the ca.pem, cert.pem and key.pem inside /etc/docker/ssl/
After that I ran the following command:

docker exec -ti ucp-kv curl --cacert /etc/docker/ssl/ca.pem --cert /etc/docker/ssl/cert.pem --key /etc/docker/ssl/key.pem https://192.168.123.14:12379/v2/keys/interlock/v1/config -XPUT -d
value=‘listenAddr=":8080"
dockerURL=“tcp://192.168.123.14:2376"
tlsCaCert=”/certs/ca.pem"
tlsCert="/certs/cert.pem"
tlsKey="/certs/key.pem"
[[Extensions]]
Name=“nginx"
ConfigPath=”/etc/conf/nginx.conf"
PidPath="/etc/conf/nginx.pid"
BackendOverrideAddress="“
ConnectTimeout=5000
ServerTimeout=10000
ClientTimeout=10000
MaxConn=1024
Port=80
SyslogAddr=”"
NginxPlusEnabled=false
AdminUser=“admin"
AdminPass=”“
SSLCertPath=”“
SSLCert=”“
SSLPort=443
SSLOpts=”"
User="www-data"
WorkerProcesses=2
RLimitNoFile=65535
ProxyConnectTimeout=600
ProxySendTimeout=600
ProxyReadTimeout=600
SendTimeout=600
SSLCiphers=
"HIGH:!aNULL:!MD5"
SSLProtocols=“SSLv3 TLSv1 TLSv1.1 TLSv1.2”’

After I ran that command I get a jsonlike key-value pair string, so I guess it worked ?
/certs/ca.pem etc is empty though…
I don’t remember setting up certificates and keys for any of my docker nodes (so maybe that’s the problem?)

Then, I go to the node where my load balancer will be running (192.168.123.39, boot2docker). This node is also part of the UCP cluster.
I pulled the git interlock-lbs repo

I set the CONTROLLER_IP=19.168.123.14 and did a docker-compose up -d inside (interlock-lbs/interlock-nginx).
But when I check the docker-compose logs I get the following errors:

nginx_1 | 2016/04/19 12:32:33 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/19 12:32:34 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/19 12:32:35 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/19 12:32:36 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/19 12:32:38 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:31Z” level=info msg="interlock 1.0.0 (49863fc)"
interlock_1 | time=“2016-04-19T12:32:31Z” level=debug msg="using kv: addr=etcd://192.168.123.14:12379"
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:31Z” level=debug msg="Trusting certs with subjects: [0\x1e1\x1c0\x1a\x06\x03U\x04\x03\x13\x13UCP Cluster Root CA]"
interlock_1 | time=“2016-04-19T12:32:31Z” level=debug msg="configuring TLS for KV"
nginx_1 | 2016/04/19 12:32:40 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:31Z” level=fatal msg="Near line 31 (last key parsed ‘Extensions.SSLCiphers’): Expected value but found ‘\n’ instead."
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/19 12:32:44 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:32Z” level=info msg="interlock 1.0.0 (49863fc)"
nginx_1 | 2016/04/19 12:32:50 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:32Z” level=debug msg="using kv: addr=etcd://192.168.123.14:12379"
nginx_1 | 2016/04/19 12:33:04 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-19T12:32:32Z” level=debug msg="Trusting certs with subjects: [0\x1e1\x1c0\x1a\x06\x03U\x04\x03\x13\x13UCP Cluster Root CA]"
nginx_1 | 2016/04/19 12:33:30 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)

Does anyone knows what I’m doing wrong?
My source is: https://www.docker.com/sites/default/files/RA_UCP%20Load%20Balancing-Feb%202016_1.pdf page 11 (3A Interlock and NGINX)

Thanks in advance


(Nicolaka) #2

Hey, can you try running the first docker exec command on the controller node itself ( not via the client bundle?).
If you’re doing any copy+pasting from the pdf , it could have messed up some characters so please refer to https://github.com/nicolaka/ucp-sd-ra.

When you installed UCP and performed the join commands, UCP automatically generated certs for each node. You can check by issuing the following command wherever you’re trying to deploy interlock:

ls /var/lib/docker/volumes/ucp-node-certs/_data/

you should see the ca,cert,and key.pem files.

Hope this helps,


(Hmaeck) #3

EDIT: I rolled back to the point where I had 2 working nodes and added a third one, view ‘Note3’

Thank you for the reply,
this is what I’ve done now. I logged in to the machine that has the client bundle and set the right environment variable using (env.sh). After that I ran a docker info command to make sure I was connected to the cluster.
The I ssh’d into the controller node, using docker machine. Then I ran the docker exec command from the link you provided (do I have to change anything else besides the ucp_fqdn? I changed the ucp_fqdn with the IP of the controller node, is this correct?)

This was my return value:
{“action”:“set”,“node”:{“key”:"/interlock/v1/config",“value”:“listenAddr = “:8080”\ndockerURL = “tcp://192.168.123.14:2376”\ntlsCaCert = “/certs/ca.pem”\ntlsCert = “/certs/cert.pem”\ntlsKey = “/certs/key.pem”\n\n[[Extensions]]\n Name = “nginx”\n ConfigPath = “/etc/conf/nginx.conf”\n PidPath = “/etc/conf/nginx.pid”\n BackendOverrideAddress = “”\n ConnectTimeout = 5000\n ServerTimeout = 10000\n ClientTimeout = 10000\n MaxConn = 1024\n Port = 80\n SyslogAddr = “”\n NginxPlusEnabled = false\n AdminUser = “admin”\n AdminPass = “”\n SSLCertPath = “”\n SSLCert = “”\n SSLPort = 443\n SSLOpts = “”\n User = “www-data”\n WorkerProcesses = 2\n RLimitNoFile = 65535\n ProxyConnectTimeout = 600\n ProxySendTimeout = 600\n ProxyReadTimeout = 600\n SendTimeout = 600\n SSLCiphers = “HIGH:!aNULL:!MD5”\n SSLProtocols = “SSLv3 TLSv1 TLSv1.1 TLSv1.2"”,“modifiedIndex”:130458,“createdIndex”:130458},“prevNode”:{“key”:”/interlock/v1/config",“value”:“listenAddr=”:8080"\ndockerURL=“tcp://192.168.123.14:2376”\ntlsCaCert="/certs/ca.pem"\ntlsCert="/certs/cert.pem"\ntlsKey="/certs/key.pem"\n[[Extensions]]\nName=“nginx”\nConfigPath="/etc/conf/nginx.conf"\nPidPath="/etc/conf/nginx.pid"\nBackendOverrideAddress=""\nConnectTimeout=5000\nServerTimeout=10000\nClientTimeout=10000\nMaxConn=1024\nPort=80\nSyslogAddr=""\nNginxPlusEnabled=false\nAdminUser=“admin”\nAdminPass=""\nSSLCertPath=""\nSSLCert=""\nSSLPort=443\nSSLOpts=""\nUser=“www-data”\nWorkerProcesses=2\nRLimitNoFile=65535\nProxyConnectTimeout=600\nProxySendTimeout=600\nProxyReadTimeout=600\nSendTimeout=600\nSSLCiphers=\n"HIGH:!aNULL:!MD5"\nSSLProtocols=“SSLv3 TLSv1 TLSv1.1 TLSv1.2"”,“modifiedIndex”:101724,“createdIndex”:101724}}

Then I exited the ssh to that node and ssh’d to the load balancing node via docker-machine. There I ran the command you provided: ls /var/lib/docker/volumes/ucp-node-certs/_data/
and it returned the ca.pem, the cert.pem eand the key.pem

I ran te docker-compose command on the loadbalancing node to make sure that it was installed and it is.
Then I cloned the github repo to my loadbalancing node and ran the following command: export CONTROLLER_IP=the-ip-of-the-controller-node
Followed by the command: docker-compose up -d , inside the interlock-lbs/interlock-nginx directory.
I get the message the interlock and the nginx are getting created. But when I run docker-compose logs after that command I still got the same errors :frowning:

nginx_1 | 2016/04/21 06:38:43 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 06:38:44 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 06:38:45 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 06:38:46 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 06:38:47 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 06:38:49 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-21T06:38:41Z” level=info msg="interlock 1.0.0 (49863fc)"
nginx_1 | 2016/04/21 06:38:53 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="using kv: addr=etcd://192.168.123.14:12379"
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="Trusting certs with subjects: [0\x1e1\x1c0\x1a\x06\x03U\x04\x03\x13\x13UCP Cluster Root CA]"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="configuring TLS for KV"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="using tls for communication with docker"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="docker client: url=tcp://192.168.123.14:2376"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="loading extension: name=nginx configpath=/etc/conf/nginx.conf"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="starting event handling"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="checking to reload"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg=reloading
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="updating load balancers"
interlock_1 | time=“2016-04-21T06:38:41Z” level=error msg="Get https://192.168.123.14:2376/v1.15/containers/json?all=0&size=0: remote error: bad certificate"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="reload duration: 34.80ms"
interlock_1 | time=“2016-04-21T06:38:41Z” level=error msg="event stream fail; attempting to reconnect"
interlock_1 | time=“2016-04-21T06:38:41Z” level=info msg="waiting for event stream to become ready"
interlock_1 | time=“2016-04-21T06:38:41Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:42Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:43Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:44Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:45Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:46Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:47Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:48Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:49Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:50Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:51Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:52Z” level=debug msg="event stream not yet ready; retrying"
interlock_1 | time=“2016-04-21T06:38:54Z” level=debug msg=“event stream not yet ready; retrying”

I think I’m getting close, I found someone else with the same problem (he’s also using boot2docker)

But My docker compose file for interlock looks like this:

version : “2”

services:

interlock:
image: ehazlett/interlock:1.0.0
command: -D run --discovery etcd://$CONTROLLER_IP:12379 --discovery-tls-ca-cert /kvcerts/ca.pem --discovery-tls-cert /kvcerts/cert.pem --discovery-tls-key /kvcerts/key.pem
volumes:
- ucp-node-certs:/kvcerts
- ucp-node-certs:/certs
- nginx:/etc/conf
network_mode: "bridge"
restart: always
environment:
- affinity:container!=interlock
nginx:
image: nginx
command: nginx -g “daemon off;” -c /etc/conf/nginx.conf
ports:
- 80:80
- 443:443
labels:
- "interlock.ext.name=nginx"
volumes:
- nginx:/etc/conf
depends_on:
- interlock
network_mode: "bridge"
restart: always
environment:
- affinity:container!=nginx
volumes:
nginx:
ucp-node-certs:
external: true

I changed the beginning to this:

command: -D run --discovery etcd://$CONTROLLER_IP:12379 --discovery-tls-ca-cert /kvcerts/ca.pem --discovery-tls-cert /kvcerts/server.pem --discovery-tls-key /kvcerts/server-key.pem
volumes:
- /var/lib/boot2docker:/kvcerts
- /var/lib/boot2docker:/certs

I have no idea if I’m getting closer or made it worse :frowning:

The docker-compose logs look now like this:

nginx_1 | 2016/04/21 07:30:58 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 07:30:59 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 07:31:00 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | 2016/04/21 07:31:01 [emerg] 1#1: open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
nginx_1 | nginx: [emerg] open() “/etc/conf/nginx.conf” failed (2: No such file or directory)
interlock_1 | time=“2016-04-21T07:30:55Z” level=info msg="interlock 1.0.0 (49863fc)"
interlock_1 | time=“2016-04-21T07:30:55Z” level=debug msg="using kv: addr=etcd://192.168.123.14:12379"
interlock_1 | time=“2016-04-21T07:30:55Z” level=debug msg="Trusting certs with subjects: [0\x0f1\r0\v\x06\x03U\x04\n\x13\x04root]"
interlock_1 | time=“2016-04-21T07:30:55Z” level=debug msg="configuring TLS for KV"
interlock_1 | time=“2016-04-21T07:30:55Z” level=fatal msg="client: etcd cluster is unavailable or misconfigured"
interlock_1 | time=“2016-04-21T07:30:56Z” level=info msg="interlock 1.0.0 (49863fc)"
interlock_1 | time=“2016-04-21T07:30:56Z” level=debug msg="using kv: addr=etcd://192.168.123.14:12379"
interlock_1 | time=“2016-04-21T07:30:56Z” level=debug msg="Trusting certs with subjects: [0\x0f1\r0\v\x06\x03U\x04\n\x13\x04root]"
interlock_1 | time=“2016-04-21T07:30:56Z” level=debug msg="configuring TLS for KV"
interlock_1 | time=“2016-04-21T07:30:56Z” level=fatal msg="client: etcd cluster is unavailable or misconfigured"
interlock_1 | time=“2016-04-21T07:30:57Z” level=info msg=“interlock 1.0.0 (49863fc)”

I also tried /var/lib/docker/volumes/ucp-node-certs/_data/ as a volume for the keys and certificates but this also results in an

level=error msg=“Get https://192.168.123.14:2376/v1.15/containers/json?all=0&size=0: remote error: bad certificate”

Note: I tripple checked that the ucp-kv (etcd) was correctly exposed at port (192.168.123.14:12379) and the ip address is reachable from the loadbalancing node.

Note2: it is possible but I’m not sure anymore that on the controller node I copied the ca.pem cert.pem and cert.pem from somewhere else to /etc/docker/ssl/ because there weren’t any certificates. I don’t know if this is a problem but I guess it is.

drwxr-xr-x 2 root root 4096 Apr 19 12:23 ./
drwxr-xr-x 4 root root 4096 Apr 19 12:22 …/
-rw-r–r-- 1 docker staff 3684 Apr 19 12:21 ca.pem
-rw-r–r-- 1 docker staff 1708 Apr 19 12:21 cert.pem
-rw------- 1 docker staff 1675 Apr 19 12:21 key.pem

The node exists much longer than the 19 of april so it must have been me who has done it. Should I replace them with something else?

NOTE 3:

I just rolled back my vm’s which are running the docker nodes. I returned to the point where I had 2 fully functioning nodes. I’ve just added the third (for the loadbalancing). What should I run :slight_smile:

Again I tried everything :frowning: I even extracted the ca.pem , cert.pem and key.pem from the ucp-kv (on node1) and mounted it as the volume for in the docker-compose.yml for nginx and interlock. I really don’t know which certificates I should be using for the interlock container.


(Nicolaka) #4

Hey, maybe it’s better if you can try the new config I loaded using interlock 1.1. Instead of using the k/v store for interlock configs, we’re just using a local toml file. So only thing you need to do is get the interlock-lbs repo on the node you wish to run interlock on, then alter the config.toml file to update the $UCP_FQDN, then run docker-compose -f docker-compose-v1.1.yml up -d. Please let me know if it works!!!

ps: I know i didn’t answer your way per say, however i think interlock 1.1 + config.toml mode is much better option. I’m working on updating the docs.


(Hmaeck) #5

NOTE: I figured it out, view NOTE3

Okay I will try that, fingers crossed, thanks for helping me :slight_smile:

Edit:
No luck :frowning:

This is what I get when I follow the steps you provided :frowning:

docker@esxdockerengine3:~/interlock-lbs/interlock-nginx$ docker-compose logs
Attaching to interlocknginx_nginx_1, interlocknginx_interlock_1
interlock_1  | time="2016-04-22T06:10:46Z" level=info msg="interlock 1.1.0 (8a68c99)"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="loading config from: /bin/config.toml"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="using tls for communication with docker"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="docker client: url=tcp://192.168.123.14:2376"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="loading extension: name=nginx"
interlock_1  | time="2016-04-22T06:10:46Z" level=info msg="interlock node: id=6b91eeb7d3c963112d2621102884495be929f3b88c9e861d9d5d0d0f6705f15c" ext=lb
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="starting event handling"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="event received: status=interlock-start id=0 type= action="
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="notifying extension: lb"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="triggering reload" ext=lb
interlock_1  | time="2016-04-22T06:10:46Z" level=error msg="event stream fail; attempting to reconnect"
interlock_1  | time="2016-04-22T06:10:46Z" level=info msg="waiting for event stream to become ready"
interlock_1  | time="2016-04-22T06:10:46Z" level=debug msg="event stream not yet ready; retrying"
interlock_1  | time="2016-04-22T06:10:47Z" level=debug msg="reaping key: reload"
interlock_1  | time="2016-04-22T06:10:47Z" level=debug msg="triggering reload from cache" ext=lb
interlock_1  | time="2016-04-22T06:10:47Z" level=debug msg="checking to reload" ext=lb
interlock_1  | time="2016-04-22T06:10:47Z" level=debug msg="updating load balancers" ext=lb
interlock_1  | time="2016-04-22T06:10:47Z" level=error msg="Get https://192.168.123.14:2376/v1.15/containers/json?all=0&size=0: remote error: bad certificate" ext=lb
interlock_1  | time="2016-04-22T06:10:47Z" level=debug msg="event stream not yet ready; retrying"
interlock_1  | time="2016-04-22T06:10:48Z" level=debug msg="event stream not yet ready; retrying"
interlock_1  | time="2016-04-22T06:10:49Z" level=debug msg="event stream not yet ready; retrying"
interlock_1  | time="2016-04-22T06:10:50Z" level=debug msg="event stream not yet ready; retrying"

It seems like there is still something wrong with the certificates to sign in. nginx itself stopped complaining, which is nice :slight_smile:, we made some progress today.

Should I maybe run docker-machine -D regenerate-certs on a machine? And if so, on which machine (the loadbalancer, the controller or the otther node, or all three?). And also, will this brake something (my local dtr for example, because I remember importing an ucp key inside dtr, but that one is probably different from the node certificates, right ?) ?

Also, here is my docker info if this is important:

[root@localhost ~]# docker info
Containers: 22
 Running: 18
 Paused: 0
 Stopped: 4
Images: 41
Server Version: swarm/1.1.3
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
 esxdockerengine1: 192.168.123.14:12376
  └ Status: Healthy
  └ Containers: 8
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 64.42 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.1.19-boot2docker, location=on_premise_BE, operatingsystem=Boot2Docker 1.10.3 (TCL 6.4.1); master : 625117e - Thu Mar 10 22:09:02 UTC 2016, provider=vmwarevsphere, storagedriver=aufs, target=apps, type=controllers
  └ Error: (none)
  └ UpdatedAt: 2016-04-22T06:49:01Z
 esxdockerengine2: 192.168.123.15:12376
  └ Status: Healthy
  └ Containers: 6
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 64.42 GiB
  └ Labels: executiondriver=native-0.2, kernelversion=4.1.19-boot2docker, location=on_premise_BE, operatingsystem=Boot2Docker 1.10.3 (TCL 6.4.1); master : 625117e - Thu Mar 10 22:09:02 UTC 2016, provider=vmwarevsphere, storagedriver=aufs, target=apps, type=secondary
  └ Error: (none)
  └ UpdatedAt: 2016-04-22T06:49:35Z
 esxdockerengine3: 192.168.123.39:12376
  └ Status: Healthy
  └ Containers: 8
  └ Reserved CPUs: 0 / 8
  └ Reserved Memory: 0 B / 64.42 GiB
  └ Labels: executiondriver=, kernelversion=4.1.19-boot2docker, location=on_premise_BE, operatingsystem=Boot2Docker 1.11.0 (TCL 7.0); HEAD : 32ee7e9 - Wed Apr 13 20:06:49 UTC 2016, provider=vmwarevsphere, storagedriver=aufs, target=loadbalancer, type=loadbalancing
  └ Error: (none)
  └ UpdatedAt: 2016-04-22T06:49:10Z
Cluster Managers: 1
 192.168.123.14: Healthy
  └ Orca Controller: https://192.168.123.14:443
  └ Swarm Manager: tcp://192.168.123.14:3376
  └ KV: etcd://192.168.123.14:12379
Plugins:
 Volume:
 Network:
Kernel Version: 4.1.19-boot2docker
Operating System: linux
Architecture: amd64
CPUs: 24
Total Memory: 193.3 GiB

Here is an image with all my running containers should that be important:

Note2: I catted the certificates/keys inside the interlock container to make sure they’re there and they’re.

docker exec -ti b8fe22e1ed3a cat /certs/key.pem
docker exec -ti b8fe22e1ed3a cat /certs/ca.pem
docker exec -ti b8fe22e1ed3a cat /certs/cert.pem

With what should I compare these certificates/keys to make sure they’re valid?

NOTE: I regenerated the certificates on the controller node and the loadbalancing node. No luck, using the command
docker-machine -D regenerate-certs esxdockerengine1
docker-machine -D regenerate-certs esxdockerengine3

Note2: I catted the certificates/keys inside the interlock container to make sure they are there, and they’re.
Can I compare them with something so I know they are valid?

Note3: It WORKED!!!

I hade to change some settings in the config.toml

ListenAddr = ":8080"
DockerURL = "tcp://192.168.123.14:2376"
TLSCACert = "/certs/ca.pem"
TLSCert = “/certs/server.pem” <---------- changed
TLSKey = “/certs/server-key.pem” <------ changed

[[Extensions]]
Name = "nginx"
ConfigPath = "/etc/nginx/nginx.conf"
PidPath = "/etc/nginx/nginx.pid"
MaxConn = 1024
Port = 80

and also the docker-compose-v1.1.yml
version : “2”

services:

interlock:
    image: ehazlett/interlock:1.1.0
    command: -D run -c /bin/config.toml
    ports:
        - 8080
    volumes:
        - ./config.toml:/bin/config.toml
        - /var/run/docker.sock:/var/run/docker.sock
        - /var/lib/boot2docker:/certs       <----------------------- changed
    restart: always
    network_mode: "bridge"
nginx:
    image: nginx:latest
    entrypoint: nginx
    command: -g "daemon off;" -c /etc/nginx/nginx.conf
    ports:
      - 80:80
    labels:
      - "interlock.ext.name=nginx"
    depends_on:
      - interlock
    network_mode: "bridge"
    restart: always

volumes:
ucp-node-certs:
external: true

This is probably because I use boot2docker, please consider adding this to the documentation.
When I now run docker-compose logs, I no longer see errors. But I noticed that I only see when events happen on the controller node, so when I restart a container on the controller node, I see this in docker-compose logs but if I restart a container on the loadbalancing node, I don’t see it in the logs. I don’t think this is normal. But I find it very strange that it doesn’t detect my ‘application’ that I’ve built with docker compose with the interlock.hostname and interlock.domain label. And I’ve got only one controller :octopus:
Anyone got an idea?

I started a new topic in case somebody else got the same problem: Docker interlock not detecting events apart from controller node


(Steve Flinchbaugh) #6

Hi @nicolaka, I tried following this using the config.toml file. I’m using only one controller for simplicity’s sake (I’ve also tried 3 UCP controllers with an AWS ELB in front, and using the ELB’s DNS as my $UCP_FQDN – similar results). Therefore, my $UCP_FQDN is simply that UCP controller’s private IP. From one of my UCP nodes, I updated the DockerURL in config.toml and also exported CONTROLLER_IP to that controller’s private IP. I ran docker-compose -f docker-compose-v1.1.yml up -d and am getting hung up.

The problem seems to be with Interlock and Nginx starting up. docker-compose logs yields:

[centos@ucp-node interlock-nginx]$ docker-compose logs Attaching to interlocknginx_nginx_1, interlocknginx_interlock_1 interlock_1 | time="2016-05-06T16:43:23Z" level=info msg="interlock 1.1.0 (8a68c99)" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="loading config from: /bin/config.toml" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="using tls for communication with docker" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="docker client: url=tcp://UCP.CONTROLLER.PRIVATE.IP:2376" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="loading extension: name=nginx" interlock_1 | time="2016-05-06T16:43:23Z" level=info msg="interlock node: id=a38b85cdffa1a438e9309feeed8f4516d55eec7fa9ed524efc9a277df93d1550" ext=lb interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="starting event handling" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="event received: status=interlock-start id=0 type= action=" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="notifying extension: lb" interlock_1 | time="2016-05-06T16:43:23Z" level=debug msg="triggering reload" ext=lb interlock_1 | time="2016-05-06T16:43:24Z" level=debug msg="reaping key: reload" interlock_1 | time="2016-05-06T16:43:24Z" level=debug msg="triggering reload from cache" ext=lb interlock_1 | time="2016-05-06T16:43:24Z" level=debug msg="checking to reload" ext=lb interlock_1 | time="2016-05-06T16:43:24Z" level=debug msg="updating load balancers" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="generating proxy config" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="proxy config path: /etc/nginx/nginx.conf" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="saving proxy config" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="signaling reload" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="triggering proxy network cleanup" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=info msg="reload duration: 157.28ms" ext=lb interlock_1 | time="2016-05-06T16:43:25Z" level=debug msg="checking to remove proxy containers from networks" ext=lb

Any ideas? I’m getting no logs from the Nginx container, so I’m assuming that Interlock is not finished its startup/config process. Appreciate the help!


(Nicolaka) #7

Looks like it got an event from UCP/Swarm. Did you alter the docker-compose-v1.1.yml file?


(Steve Flinchbaugh) #8

No, I did not. Using the docker-compose-v1.1.yml file did get me further than curling the config into the UCP k-v, though. I did manage to get the HAProxy implementation working, so for now I am happy with that. Still do not know why Nginx is having issues. Thank you for your writing the walkthrough and for your help!


(Hmaeck) #9

Can you maybe post the output of the docker info command?
Also make sure the swarm controller is running at 2376, mine was running at 3376. (boot2docker)


(Gracco Guimaraes) #10

Guys I’m the only one here that is having this issue?
root@ddc-controller-poc:/var/lib/docker/volumes/ucp-node-certs/_data# docker exec -ti ucp-kv curl \

–cacert /etc/docker/ssl/ca.pem
–cert /etc/docker/ssl/cert.pem
–key /etc/docker/ssl/key.pem
https://$UCP_FQDN:12379/v2/keys/interlock/v1/config -XPUT -d
value=‘listenAddr = “:8080” dockerURL = "tcp://$UCP_FQDN:2376"
tlsCaCert = "/certs/ca.pem"
tlsCert = "/certs/cert.pem"
tlsKey = “/certs/key.pem”
[[extensions]]
name = "haproxy"
configPath = "/usr/local/etc/haproxy/haproxy.cfg"
pidPath = "/usr/local/etc/haproxy/haproxy.pid"
sslCert = ""
maxConn = 1024
port = 80
sslPort = 443
adminUser = "admin"
adminPass = “”’

exec: “curl”: executable file not found in $PATH

My ucp-kv doesn’t have curl binary, follow my docker ps
root@ddc-controller-poc:/var/lib/docker/volumes/ucp-node-certs/_data# docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
a7566ad65071 docker/ucp-controller:1.1.0 “/bin/controller serv” 25 hours ago Up 23 hours 0.0.0.0:443->8080/tcp ucp-controller
0c1a4f9e2a2f docker/ucp-auth:1.1.0 "/usr/local/bin/enzi " 25 hours ago Up 23 hours 0.0.0.0:12386->4443/tcp ucp-auth-worker
1c4b02013bd8 docker/ucp-auth:1.1.0 "/usr/local/bin/enzi " 25 hours ago Up 23 hours 0.0.0.0:12385->4443/tcp ucp-auth-api
976381bcbde1 docker/ucp-auth-store:1.1.0 “/usr/local/bin/rethi” 25 hours ago Up 23 hours 0.0.0.0:12383-12384->12383-12384/tcp ucp-auth-store
d36efb77b12d docker/ucp-cfssl:1.1.0 “/bin/cfssl serve -ad” 25 hours ago Up 23 hours 8888/tcp, 0.0.0.0:12381->12381/tcp ucp-cluster-root-ca
0a6f126f0b26 docker/ucp-cfssl:1.1.0 “/bin/cfssl serve -ad” 25 hours ago Up 23 hours 8888/tcp, 0.0.0.0:12382->12382/tcp ucp-client-root-ca
65ff9ce846a8 docker/ucp-swarm:1.1.0 “/swarm manage --tlsv” 25 hours ago Up 23 hours 0.0.0.0:2376->2375/tcp ucp-swarm-manager
64db74e8b551 docker/ucp-swarm:1.1.0 “/swarm join --discov” 25 hours ago Up 23 hours 2375/tcp ucp-swarm-join
84e9bb254897 docker/ucp-proxy:1.1.0 “/bin/run” 25 hours ago Up 23 hours 0.0.0.0:12376->2376/tcp ucp-proxy
4457d237f4f3 docker/ucp-etcd:1.1.0 “/bin/etcd --data-dir” 25 hours ago Up 23 hours 2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp ucp-kv

I am using Ubuntu 14.04 - Docker version 1.10.3-cs3, build 6df5588

Thanks