# Problem adding new node to cluster (certificate problem) **URL:** \ **Category:** Swarm\ **Tags:** ucp\ **Created:** [May 3, 2016, 12:17pm UTC](https://forums.docker.com/t/problem-adding-new-node-to-cluster-certificate-problem/10776 "2016-05-03T12:17:57Z")\ **Posts on this page:** 1\ **Page:** 1
**Author:** ![hmaeck](https://avatars.discourse-cdn.com/v4/letter/h/e9c0ed/32.png) [@hmaeck](https://forums.docker.com/u/hmaeck)\ **Post date:** [May 3, 2016, 12:17pm UTC](https://forums.docker.com/t/problem-adding-new-node-to-cluster-certificate-problem/10776/1 "2016-05-03T12:17:57Z")
Hello, I’ve got a setup with 3 nodes (3 x boot2docker on vmware) and I want to add another node using the generic driver. I ssh’d into the new node that I’ve got provisioned iwth docker using docker-machine (base-os is debian). Then I ran the following command to add the new node to swarm. (swamr manager is located at 192.168.123.14) > [root@localhost docker]# docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock --name ucp docker/ucp join -i --host-address $(docker-machine ip devstackdockerengine1) --fresh-install -D > DEBU[0000] Starting join > DEBU[0000] Verifying docker.sock > DEBU[0000] Connecting to docker unix:///var/run/docker.sock > DEBU[0000] Checking for compatible kernel version > DEBU[0000] Kernel version 3.16.0-4-amd64 is compatible > DEBU[0000] Checking for compatible engine version > DEBU[0000] Engine version 1.11.1 is compatible > DEBU[0000] Looking for container ucp-phase2 > DEBU[0000] Container ucp-phase2 not found: Not found > DEBU[0000] Looking for container ucp > DEBU[0000] Looking for container ucp-controller > DEBU[0000] Container ucp-controller not found: Not found > DEBU[0000] Looking for container ucp-swarm-manager > DEBU[0000] Container ucp-swarm-manager not found: Not found > DEBU[0000] Looking for container ucp-swarm-join > DEBU[0000] Container ucp-swarm-join not found: Not found > DEBU[0000] Looking for container ucp-kv > DEBU[0000] Container ucp-kv not found: Not found > DEBU[0000] Looking for container ucp-proxy > DEBU[0000] Looking for container ucp-client-root-ca > DEBU[0000] Container ucp-client-root-ca not found: Not found > DEBU[0000] Looking for container ucp-cluster-root-ca > DEBU[0000] Container ucp-cluster-root-ca not found: Not found > DEBU[0000] Validating base system meets minimum requirements > DEBU[0000] Your system meets minimum memory requirements: 7.72 GB \>= 1.50 GB > DEBU[0001] Your system meets minimum storage requirements: 11.58 GB \>= 3.00 GB > Please enter the URL to your UCP server: [https://192.168.123.14](https://192.168.123.14) > DEBU[0020] Performing TOFU check on 192.168.123.14:443 > UCP server [https://192.168.123.14](https://192.168.123.14) > Subject: ucp > Issuer: UCP Client Root CA > SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6 > Do you want to trust this server and proceed with the join? (y/n): y > DEBU[0022] User accepted TLS Fingerprint “SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6”, proceeding > Please enter your UCP Admin username: admin > Please enter your UCP Admin password: > DEBU[0028] Checking for images > INFO[0028] All required images are present > DEBU[0028] Local Name: devstackdockerengine1 > DEBU[0028] Host Address: 192.168.123.1 > WARN[0028] None of the hostnames we’ll be using in the UCP certificates [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] contain a domain component. Your generated certs may fail TLS validation unless you only use one of these shortnames or IPs to connect. You can use the --san flag to add more aliases > You may enter additional aliases (SANs) now or press enter to proceed with the above list. > Additional aliases: > DEBU[0072] User entered: > DEBU[0072] Hostnames: [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] > DEBU[0072] Re-using existing volume ucp-node-certs > DEBU[0072] Launching phase 2 > DEBU[0074] Launching phase 2 with: [join -i --host-address 192.168.123.1 --fresh-install -D] (cac5fd0b2bcad85bb2e7887528e9e524cb74aeb5261bea713376d3b9d86cd255) > DEBU[0066] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379] > DEBU[0066] Applicable SwarmArgs: [–discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379]DEBU[0000] Starting join > DEBU[0000] Verifying docker.sock > DEBU[0000] Connecting to docker unix:///var/run/docker.sock > DEBU[0000] Checking for compatible kernel version > DEBU[0000] Kernel version 3.16.0-4-amd64 is compatible > DEBU[0000] Checking for compatible engine version > DEBU[0000] Engine version 1.11.1 is compatible > DEBU[0000] Looking for container ucp-phase2 > DEBU[0000] Looking for container ucp-controller > DEBU[0000] Container ucp-controller not found: Not found > DEBU[0000] Looking for container ucp-swarm-manager > DEBU[0000] Container ucp-swarm-manager not found: Not found > DEBU[0000] Looking for container ucp-swarm-join > DEBU[0000] Container ucp-swarm-join not found: Not found > DEBU[0000] Looking for container ucp-kv > DEBU[0000] Container ucp-kv not found: Not found > DEBU[0000] Looking for container ucp-proxy > DEBU[0000] Looking for container ucp-client-root-ca > DEBU[0000] Container ucp-client-root-ca not found: Not found > DEBU[0000] Looking for container ucp-cluster-root-ca > DEBU[0000] Container ucp-cluster-root-ca not found: Not found > DEBU[0000] Local Name: devstackdockerengine1 > DEBU[0000] Host Address: 192.168.123.1 > DEBU[0000] Hostnames: [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] > INFO[0000] This engine will join UCP and advertise itself with host address 192.168.123.1 - If this is incorrect, please specify an alternative address with the ‘–host-address’ flag > INFO[0000] Verifying your system is compatible with UCP > INFO[0000] Removing old UCP containers > DEBU[0000] Detected bootstrapper, omitting > DEBU[0000] Found UCP:4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J container [/ucp-proxy] (9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c) > DEBU[0000] Stopping container 9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c > DEBU[0007] Removing container 9b4ccc2e5a7b9d4c35ca83f4b5f7f42e4623bd6c1b7c9a015902e1433e43933c > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_kv > DEBU[0007] Cleaning stale data from ucp-config > DEBU[0007] Cleaning stale data from /var/lib/docker/discovery\_certs > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_client\_root\_ca > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_controller\_server\_certs > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_controller\_client\_certs > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_kv\_certs > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_node\_certs > DEBU[0007] Cleaning stale data from /var/lib/docker/ucp/ucp\_cluster\_root\_ca > DEBU[0007] Attempting connection with trusted fingerprint: SHA1 Fingerprint=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6 > DEBU[0008] SHA1 FINGERPRINT=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6 ~= SHA1 FINGERPRINT=E3:46:28:F9:6A:92:6E:F2:93:C0:EE:99:67:59:AE:54:45:6C:90:C6 > DEBU[0008] Server cert(s) passed TOFU tests > DEBU[0008] CSR Generated for “swarm node” with hostnames [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] > DEBU[0050] Joining UCP ID: 4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J > DEBU[0050] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379] > DEBU[0050] Raw SwarmArgs: [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379] > DEBU[0050] [join --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --advertise 192.168.123.39:12376 --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379] > DEBU[0050] Applicable SwarmArgs: [–discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem --discovery-opt kv.cacertfile=/etc/docker/ssl/ca.pem --discovery-opt kv.certfile=/etc/docker/ssl/cert.pem --discovery-opt kv.keyfile=/etc/docker/ssl/key.pem etcd://192.168.123.14:12379] > DEBU[0050] CSR Generated for “discovery” with hostnames [devstackdockerengine1 127.0.0.1 172.17.0.1 192.168.123.1] > DEBU[0096] Joining UCP ID: 4AOG:QGIZ:56UD:WOAJ:EUWZ:WISV:W3UW:DZ72:LJR7:LOIS:CE7G:CO2J > INFO[0096] Starting local swarm containers > DEBU[0096] Starting docker proxy > DEBU[0111] Proxy started on 192.168.123.1:12376 > DEBU[0111] Checking for liveness of [https://192.168.123.1:12376](https://192.168.123.1:12376) > ERRO[0172] We were unable to communicate with proxy we just started at address 192.168.123.1. Did you forget to specify an alternate DNS server with the ‘–dns’ flag? If this address is incorrect, re-run the install using the ‘–host-address’ option. Run “docker logs ucp-proxy” for more details from the proxy > FATA[0172] Unable to connect to system So I ran docker logs ucp-proxy: ``` [root@localhost docker]# docker logs ucp-proxy Using TLS Listening on 2376 time="2016-05-03T11:50:02Z" level=info msg="docker proxy" time="2016-05-03T11:50:02Z" level=info msg="Configuring TLS: ca=/etc/docker/ssl/ca.pem cert=/etc/docker/ssl/cert.pem key=/etc/docker/ssl/key.pem" 2016/05/03 11:50:03 http: TLS handshake error from 192.168.123.1:37640: remote error: bad certificate 2016/05/03 11:50:04 http: TLS handshake error from 192.168.123.1:37641: remote error: bad certificate 2016/05/03 11:50:04 http: TLS handshake error from 192.168.123.1:37642: remote error: bad certificate 2016/05/03 11:50:05 http: TLS handshake error from 192.168.123.1:37643: remote error: bad certificate 2016/05/03 11:50:05 http: TLS handshake error from 192.168.123.1:37644: remote error: bad certificate ``` I checked /etc/docker/ssl does not exist. What should I do? EDIT: ( [https://forums.docker.com/t/cant-spawn-ucp-0-7-1-0-8-0-1-0-0-1-0-1-on-centos-7-2-unable-to-communicate-with-proyx-at-https-host-12376/5852/15](https://forums.docker.com/t/cant-spawn-ucp-0-7-1-0-8-0-1-0-0-1-0-1-on-centos-7-2-unable-to-communicate-with-proyx-at-https-host-12376/5852/15)) I tried the following (as suggested in the topic above): > cd /var/lib/docker/volumes/ucp-node-certs/\_data > curl -v --cacert ca.pem --cert cert.pem --key key.pem [https://192.168.123.1:12376/info](https://192.168.123.1:12376/info) And that worked, I was able to communicate with the UCP-proxy container Then I tried this: > docker run -it --rm -v ucp-node-certs:/certs mbentley/curl -v --cacert /certs/ca.pem --cert /certs/cert.pem --key /certs/key.pem [https://192.168.123.1:12376/info](https://192.168.123.1:12376/info) This was the output I got: > - Trying 192.168.123.1… > - Connected to 192.168.123.1 (192.168.123.1) port 12376 (#0) > - ALPN, offering http/1.1 > - Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH > - successfully set certificate verify locations: > - CAfile: /certs/ca.pem > CApath: none > - TLSv1.2 (OUT), TLS header, Certificate Status (22): > - TLSv1.2 (OUT), TLS handshake, Client hello (1): > - TLSv1.2 (IN), TLS handshake, Server hello (2): > - TLSv1.2 (IN), TLS handshake, Certificate (11): > - TLSv1.2 (IN), TLS handshake, Server key exchange (12): > - TLSv1.2 (IN), TLS handshake, Request CERT (13): > - TLSv1.2 (IN), TLS handshake, Server finished (14): > - TLSv1.2 (OUT), TLS handshake, Certificate (11): > - TLSv1.2 (OUT), TLS handshake, Client key exchange (16): > - TLSv1.2 (OUT), TLS handshake, CERT verify (15): > - TLSv1.2 (OUT), TLS change cipher, Client hello (1): > - TLSv1.2 (OUT), TLS handshake, Finished (20): > - TLSv1.2 (IN), TLS alert, Server hello (2): > - error:140943FC:SSL routines:ssl3\_read\_bytes:sslv3 alert bad record mac > - Closing connection 0 > - TLSv1.2 (OUT), TLS alert, Client hello (1): > curl: (35) error:140943FC:SSL routines:ssl3\_read\_bytes:sslv3 alert bad record mac EDIT2: Hmmmm I just respawned the ucp proxy, and got this error while doing the curl: > - Hostname was NOT found in DNS cache > - Trying 192.168.123.1… > - Connected to 192.168.123.1 (192.168.123.1) port 12376 (#0) > - successfully set certificate verify locations: > - CAfile: ca.pem > CApath: /etc/ssl/certs > - SSLv3, TLS handshake, Client hello (1): > - SSLv3, TLS handshake, Server hello (2): > - SSLv3, TLS handshake, CERT (11): > - SSLv3, TLS alert, Server hello (2): > - SSL certificate problem: certificate is not yet valid > - Closing connection 0 > curl: (60) SSL certificate problem: certificate is not yet valid > More details here: [curl - SSL CA Certificates](http://curl.haxx.se/docs/sslcerts.html) > curl performs SSL certificate verification by default, using a “bundle” > of Certificate Authority (CA) public keys (CA certs). If the default > bundle file isn’t adequate, you can specify an alternate file > using the --cacert option. > If this HTTPS server uses a certificate signed by a CA represented in > the bundle, the certificate verification probably failed due to a > problem with the certificate (it might be expired, or the name might > not match the domain name in the URL). > If you’d like to turn off curl’s verification of the certificate, use > the -k (or --insecure) option. There is probably something wrong with the time settings. Edit3: I solved it setting the clock at the right time. This was really a stupid problem lol. But the error message ‘bad certificate didn’t really work’. I would have prefered something like ‘certificate not valid yet’ But now I’ve got another problem. The node joined the swarm but it kicked out one of my boot2docker nodes :s Edit4: I had to add the node again. Very strange but now it works 🙂