Problem adding new node to cluster (certificate problem)

Hello,
I’ve got a setup with 3 nodes (3 x boot2docker on vmware) and I want to add another node using the generic driver. I ssh’d into the new node that I’ve got provisioned iwth docker using docker-machine (base-os is debian).
Then I ran the following command to add the new node to swarm. (swamr manager is located at 192.168.123.14)

[root@localhost docker]# docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock --name ucp docker/ucp join -i --host-address $(docker-machine ip devstackdockerengine1) --fresh-install -D
DEBU[0000] Starting join
DEBU[0000] Verifying docker.sock
DEBU[0000] Connecting to docker unix:///var/run/docker.sock
DEBU[0000] Checking for compatible kernel version
DEBU[0000] Kernel version 3.16.0-4-amd64 is compatible
DEBU[0000] Checking for compatible engine version
DEBU[0000] Engine version 1.11.1 is compatible
DEBU[0000] Looking for container ucp-phase2
DEBU[0000] Container ucp-phase2 not found: Not found
DEBU[0000] Looking for container ucp
DEBU[0000] Looking for container ucp-controller
DEBU[0000] Container ucp-controller not found: Not found
DEBU[0000] Looking for container ucp-swarm-manager
DEBU[0000] Container ucp-swarm-manager not found: Not found
DEBU[0000] Looking for container ucp-swarm-join
DEBU[0000] Container ucp-swarm-join not found: Not found
DEBU[0000] Looking for container ucp-kv
DEBU[0000] Container ucp-kv not found: Not found
DEBU[0000] Looking for container ucp-proxy
DEBU[0000] Looking for container ucp-client-root-ca
DEBU[0000] Container ucp-client-root-ca not found: Not found
DEBU[0000] Looking for container ucp-cluster-root-ca
DEBU[0000] Container ucp-cluster-root-ca not found: Not found
DEBU[0000] Validating base system meets minimum requirements
DEBU[0000] Your system meets minimum memory requirements: 7.72 GB >= 1.50 GB
DEBU[0001] Your system meets minimum storage requirements: 11.58 GB >= 3.00 GB
Please enter the URL to your UCP server: https://192.168.123.14
DEBU[0020] Performing TOFU check on 192.168.123.14:443
UCP server https://192.168.123.14
Subject: ucp
Issuer: UCP Client Root CA
SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6
Do you want to trust this server and proceed with the join? (y/n): y
DEBU[0022] User accepted TLS Fingerprint “SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6”, proceeding
Please enter your UCP Admin username: admin
Please enter your UCP Admin password:
DEBU[0028] Checking for images
INFO[0028] All required images are present
DEBU[0028] Local Name: devstackdockerengine1
DEBU[0028] Host Address: 192.168.123.1
WARN[0028] None of the hostnames we’ll be using in the UCP certificates [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] contain a domain component. Your generated certs may fail TLS validation unless you only use one of these shortnames or IPs to connect. You can use the --san flag to add more aliases

You may enter additional aliases (SANs) now or press enter to proceed with the above list.
Additional aliases:
DEBU[0072] User entered:

DEBU[0072] Hostnames: [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1]
DEBU[0072] Re-using existing volume ucp-node-certs
DEBU[0072] Launching phase 2
DEBU[0074] Launching phase 2 with: [join -i --host-address 192.168.123.1 --fresh-install -D] (cac5fd0b2bcad85bb2e7887528e9e524cb74aeb5261bea713376d3b9d86cd255)
DEBU[0066] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]
DEBU[0066] Applicable SwarmArgs: [–discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]DEBU[0000] Starting join
DEBU[0000] Verifying docker.sock
DEBU[0000] Connecting to docker unix:///var/run/docker.sock
DEBU[0000] Checking for compatible kernel version
DEBU[0000] Kernel version 3.16.0-4-amd64 is compatible
DEBU[0000] Checking for compatible engine version
DEBU[0000] Engine version 1.11.1 is compatible
DEBU[0000] Looking for container ucp-phase2
DEBU[0000] Looking for container ucp-controller
DEBU[0000] Container ucp-controller not found: Not found
DEBU[0000] Looking for container ucp-swarm-manager
DEBU[0000] Container ucp-swarm-manager not found: Not found
DEBU[0000] Looking for container ucp-swarm-join
DEBU[0000] Container ucp-swarm-join not found: Not found
DEBU[0000] Looking for container ucp-kv
DEBU[0000] Container ucp-kv not found: Not found
DEBU[0000] Looking for container ucp-proxy
DEBU[0000] Looking for container ucp-client-root-ca
DEBU[0000] Container ucp-client-root-ca not found: Not found
DEBU[0000] Looking for container ucp-cluster-root-ca
DEBU[0000] Container ucp-cluster-root-ca not found: Not found
DEBU[0000] Local Name: devstackdockerengine1
DEBU[0000] Host Address: 192.168.123.1
DEBU[0000] Hostnames: [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1]
INFO[0000] This engine will join UCP and advertise itself with host address 192.168.123.1 - If this is incorrect, please specify an alternative address with the ‘–host-address’ flag
INFO[0000] Verifying your system is compatible with UCP
INFO[0000] Removing old UCP containers
DEBU[0000] Detected bootstrapper, omitting
DEBU[0000] Found UCP:4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J container [/ucp-proxy] (9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c)
DEBU[0000] Stopping container 9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c
DEBU[0007] Removing container 9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_kv
DEBU[0007] Cleaning stale data from ucp-config
DEBU[0007] Cleaning stale data from /var/lib/docker/discovery_certs
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_client_root_ca
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_controller_server_certs
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_controller_client_certs
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_kv_certs
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_node_certs
DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp_cluster_root_ca
DEBU[0007] Attempting connection with trusted fingerprint: SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6
DEBU[0008] SHA1 FINGERPRINT=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6 ~= SHA1 FINGERPRINT=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6
DEBU[0008] Server cert(s) passed TOFU tests
DEBU[0008] CSR Generated for “swarm node” with hostnames [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1]
DEBU[0050] Joining UCP ID: 4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J
DEBU[0050] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]
DEBU[0050] Raw SwarmArgs: [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]
DEBU[0050] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]
DEBU[0050] Applicable SwarmArgs: [–discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]
DEBU[0050] CSR Generated for “discovery” with hostnames [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1]
DEBU[0096] Joining UCP ID: 4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J
INFO[0096] Starting local swarm containers
DEBU[0096] Starting docker proxy
DEBU[0111] Proxy started on 192.168.123.1:12376
DEBU[0111] Checking for liveness of https://192.168.123.1:12376
ERRO[0172] We were unable to communicate with proxy we just started at address 192.168.123.1. Did you forget to specify an alternate DNS server with the ‘–dns’ flag? If this address is incorrect, re-run the install using the ‘–host-address’ option. Run “docker logs ucp-proxy” for more details from the proxy
FATA[0172] Unable to connect to system

So I ran docker logs ucp-proxy:

[root@localhost docker]# docker logs ucp-proxy
Using TLS
Listening on 2376
time="2016-05-03T11:50:02Z" level=info msg="docker proxy"
time="2016-05-03T11:50:02Z" level=info msg="Configuring TLS: ca=/etc/docker/ssl/ca.pem cert=/etc/docker/ssl/cert.pem key=/etc/docker/ssl/key.pem"
2016/05/03 11:50:03 http: TLS handshake error from 192.168.123.1:37640: remote error: bad certificate
2016/05/03 11:50:04 http: TLS handshake error from 192.168.123.1:37641: remote error: bad certificate
2016/05/03 11:50:04 http: TLS handshake error from 192.168.123.1:37642: remote error: bad certificate
2016/05/03 11:50:05 http: TLS handshake error from 192.168.123.1:37643: remote error: bad certificate
2016/05/03 11:50:05 http: TLS handshake error from 192.168.123.1:37644: remote error: bad certificate

I checked /etc/docker/ssl does not exist. What should I do?

EDIT:
( https://forums.docker.com/t/cant-spawn-ucp-0-7-1-0-8-0-1-0-0-1-0-1-on-centos-7-2-unable-to-communicate-with-proyx-at-https-host-12376/5852/15)
I tried the following (as suggested in the topic above):

cd /var/lib/docker/volumes/ucp-node-certs/_data
curl -v --cacert ca.pem --cert cert.pem --key key.pem https://192.168.123.1:12376/info

And that worked, I was able to communicate with the UCP-proxy container

Then I tried this:

docker run -it --rm -v ucp-node-certs:/certs mbentley/curl -v --cacert /certs/ca.pem --cert /certs/cert.pem --key /certs/key.pem https://192.168.123.1:12376/info

This was the output I got:

  • Trying 192.168.123.1…
  • Connected to 192.168.123.1 (192.168.123.1) port 12376 (#0)
  • ALPN, offering http/1.1
  • Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
  • successfully set certificate verify locations:
  • CAfile: /certs/ca.pem
    CApath: none
  • TLSv1.2 (OUT), TLS header, Certificate Status (22):
  • TLSv1.2 (OUT), TLS handshake, Client hello (1):
  • TLSv1.2 (IN), TLS handshake, Server hello (2):
  • TLSv1.2 (IN), TLS handshake, Certificate (11):
  • TLSv1.2 (IN), TLS handshake, Server key exchange (12):
  • TLSv1.2 (IN), TLS handshake, Request CERT (13):
  • TLSv1.2 (IN), TLS handshake, Server finished (14):
  • TLSv1.2 (OUT), TLS handshake, Certificate (11):
  • TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
  • TLSv1.2 (OUT), TLS handshake, CERT verify (15):
  • TLSv1.2 (OUT), TLS change cipher, Client hello (1):
  • TLSv1.2 (OUT), TLS handshake, Finished (20):
  • TLSv1.2 (IN), TLS alert, Server hello (2):
  • error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac
  • Closing connection 0
  • TLSv1.2 (OUT), TLS alert, Client hello (1):
    curl: (35) error:140943FC:SSL routines:ssl3_read_bytes:sslv3 alert bad record mac

EDIT2:
Hmmmm I just respawned the ucp proxy, and got this error while doing the curl:

  • Hostname was NOT found in DNS cache
  • Trying 192.168.123.1…
  • Connected to 192.168.123.1 (192.168.123.1) port 12376 (#0)
  • successfully set certificate verify locations:
  • CAfile: ca.pem
    CApath: /etc/ssl/certs
  • SSLv3, TLS handshake, Client hello (1):
  • SSLv3, TLS handshake, Server hello (2):
  • SSLv3, TLS handshake, CERT (11):
  • SSLv3, TLS alert, Server hello (2):
  • SSL certificate problem: certificate is not yet valid
  • Closing connection 0
    curl: (60) SSL certificate problem: certificate is not yet valid
    More details here: curl - SSL CA Certificates

curl performs SSL certificate verification by default, using a “bundle”
of Certificate Authority (CA) public keys (CA certs). If the default
bundle file isn’t adequate, you can specify an alternate file
using the --cacert option.
If this HTTPS server uses a certificate signed by a CA represented in
the bundle, the certificate verification probably failed due to a
problem with the certificate (it might be expired, or the name might
not match the domain name in the URL).
If you’d like to turn off curl’s verification of the certificate, use
the -k (or --insecure) option.

There is probably something wrong with the time settings.

Edit3:
I solved it setting the clock at the right time. This was really a stupid problem lol. But the error message ‘bad certificate didn’t really work’. I would have prefered something like ‘certificate not valid yet’
But now I’ve got another problem. The node joined the swarm but it kicked out one of my boot2docker nodes :s

Edit4:
I had to add the node again. Very strange but now it works :slight_smile:

1 Like