Docker Community Forums

Share and learn in the Docker community.

Can't spawn ucp (0.7.1, 0.8.0, 1.0.0, 1.0.1) on CentOS 7.2 - unable to communicate with proyx at https://host:12376

ucp
docker

(Konjak) #1

Hi there,

I’ve successfully spawned ucp 0.6.0 but I can’t spawn 0.7.1.

That’s the error message:

DEBU[0076] Checking for liveness of https://host:12376
ERRO[0136] We were unable to communicate with proxy we just started at address host. Did you forget to specify an alternate DNS server with the ‘–dns’ flag? If this address is incorrect, re-run the install using the ‘–host-address’ option. Run “docker logs ucp-proxy” for more details from the proxy
FATA[0136] Unable to connect to system

DEBU[0000] Kernel version 4.4.0-1.el7.elrepo.x86_64 is compatible
DEBU[0000] Engine version 1.9.1 is compatible

host_address is correct and I can access https://host:12376. Content of that page is:

404 page not found

I guess that’s not correct. Where should I look for errors?


(TX Fan) #2

I also encountered the same issue. I tried the below steps on 4 machines within the same network. Three of them worked and one failed.

  1. Install CentOS 7 and update to latest version.
    $ sudo yum update -y $ cat /etc/redhat-release CentOS Linux release 7.2.1511 (Core)
  2. Update Linux Kernel to 4.4.0.
    $ uname -a Linux docker-master-1.wlyz.dev 4.4.0-1.el7.elrepo.x86_64 #1 SMP Sun Jan 10 21:17:16 EST 2016 x86_64 x86_64 x86_64 GNU/Linux
  3. Install UCP
    $ docker run --rm -it -v /var/run/docker.sock:/var/run/docker.sock --name ucp docker/ucp install -i
  4. One of the machine says:
    INFO[0019] Generating Swarm Root CA INFO[0033] Generating UCP Root CA INFO[0040] Deploying UCP Containers ERRO[0101] We were unable to communicate with proxy we just started at address xxx.xxx.xxx.xxx. Did you forget to specify an alternate DNS server with the '--dns' flag? If this address is incorrect, re-run the install using the '--host-address' option. Run "docker logs ucp-proxy" for more details from the proxy FATA[0101] Unable to connect to system
  5. The container logs:
    $ docker logs ucp-proxy Using TLS Listening on 2376 time="2016-01-28T10:58:15Z" level=info msg="docker proxy" time="2016-01-28T10:58:15Z" level=info msg="Configuring TLS: cert=/etc/docker/ssl/cert.pem key=/etc/docker/ssl/key.pem"
  6. And the ports are exposed correctly.
    $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES f2e37c25b93c docker/ucp-proxy:0.7.1 "/bin/run" 4 hours ago Up 4 hours 0.0.0.0:12376->2376/tcp ucp-proxy a61f6320a0eb docker/ucp-etcd:0.7.1 "/bin/etcd --data-dir" 4 hours ago Up 4 hours 2380/tcp, 4001/tcp, 7001/tcp, 0.0.0.0:12380->12380/tcp, 0.0.0.0:12379->2379/tcp ucp-kv
    GET https://xxx.xxx.xxx.xxx:12376 results in 404 page not found.

(TX Fan) #3

I resolved this issue by disabling firewalld service.

$ systemctl stop firewalld.service
$ systemctl disable firewalld.service

(Cameronp) #4

Best to follow the documentation and it will work fine. From the docs, you need to open ports in the firewall to allow it to install. you can then keep the firewall running.


(Konjak) #5

In the firewall log I can see that docker is opening the ports correctly. When the ucp startup process is checking the availability of the ports everything is fine. Also since I can open the port from the outside and get this strange 404 message I guess this is not a port/firewall issue.


(Cameronp) #6

yeah sorry i should have mentioned my response was in regards to txfan and stopping the firewall…


(Konjak) #7

Still no luck with 0.8.0.

DEBU[0128] Proxy started on host:12376
DEBU[0128] Checking for liveness of https://host:12376
ERRO[0188] We were unable to communicate with proxy we just started at address host. Did you forget to specify an alternate DNS server with the ‘–dns’ flag? If this address is incorrect, re-run the install using the ‘–host-address’ option. Run “docker logs ucp-proxy” for more details from the proxy
FATA[0188] Unable to connect to system

What’s wrong there?


(Pycloux) #8

Hi All,
I raised another issue UCP installation issue - Host not starting ... no error! (same symptom).
CentOS 7.1, Docker 1.10, UCP 0.8.
No clue, nothing relevant in the logs, same 404 from the proxy.


(Jsenon) #9

Hi all,

Have you check that all servers have time synchronized?


(Konjak) #10

What should be synchronized? There is only one server. That doesn’t make sense.


(Konjak) #11

Hi,
I am very disappointed that this problem is still not fixed in 1.0.0 and no one from the docker team considered investigating about this issue. When participating in a beta such issue reports are normally welcome.

I have a fresh install of Centos 7.2 with an image from my host (hetzner) so I can’t believe that it is a weird configuration problem on my side.

Please take a look into this.


(Vivek Saraswat) #12

Hi konjak,

Sorry about that, must have slipped past somehow. I’ll look into it and see if we can figure out what’s going on.


(Konjak) #13

Hi, just tested 1.0.1 - still the same issue.


(Vivek Saraswat) #14

Konjak, just sent you a private message to discuss offline.


(Ashbyj) #15

Same issue when installing UCP on CentOS 7.2.

$ sudo docker run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp install -i --controller-port=8443 --host-address=docker.example.com --fresh-install --debug

This happens even when I completely disable firewalld, iptables, and SELinux.

At some point last week I was able to get this going, but couldn’t reproduce exactly what fixed it. UCP stopped responding so I went to reinstall but hitting the same thing.

Here is the end of the debug output

DEBU[0049] Starting KV container
DEBU[0049] Internal KV started at etcd://docker.example.com:12379
DEBU[0049] Starting docker proxy
DEBU[0050] Proxy started on docker.example.com:12376
DEBU[0050] Checking for liveness of https://docker.example.com:12376
ERRO[0110] We were unable to communicate with proxy we just started at address docker.example.com.  Did you forget to specify an alternate DNS server with the '--dns' flag?  If this address is incorrect, re-run the install using the '--host-address' option.    Run "docker logs ucp-proxy" for more details from the proxy
FATA[0110] Unable to connect to system

and the logs

$ sudo docker logs ucp-proxy
Using TLS
Listening on 2376
time="2016-03-15T13:38:57Z" level=info msg="docker proxy"
time="2016-03-15T13:38:57Z" level=info msg="Configuring TLS: ca=/etc/docker/ssl/ca.pem cert=/etc/docker/ssl/cert.pem key=/etc/docker/ssl/key.pem"

the docker-proxy appears to be running OK

$ ps aux | grep dock
/usr/bin/docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --exec-opt native.cgroupdriver=cgroupfs
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 443 -container-ip 172.18.0.2 -container-port 443
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 80 -container-ip 172.18.0.2 -container-port 80
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 12380 -container-ip 172.17.0.2 -container-port 12380
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 12379 -container-ip 172.17.0.2 -container-port 2379
/bin/etcd --data-dir /data --name orca-kv-docker.example.com --listen-peer-urls https://0.0.0.0:12380 --listen-client-urls https://0.0.0.0:2379 --advertise-client-urls https://docker.example.com:12379 --initial-advertise-peer-urls https://docker.example.com:12380 --initial-cluster orca-kv-docker.example.com=https://docker.example.com:12380 --initial-cluster-state new --trusted-ca-file /etc/docker/ssl/ca.pem --cert-file /etc/docker/ssl/cert.pem --key-file /etc/docker/ssl/key.pem --client-cert-auth --peer-trusted-ca-file /etc/docker/ssl/ca.pem --peer-cert-file /etc/docker/ssl/cert.pem --peer-key-file /etc/docker/ssl/key.pem --peer-client-cert-auth
docker-proxy -proto tcp -host-ip 0.0.0.0 -host-port 12376 -container-ip 172.17.0.3 -container-port 2376
docker-proxy -l :2376 -ca /etc/docker/ssl/ca.pem -cert /etc/docker/ssl/cert.pem -key /etc/docker/ssl/key.pem

Seems like a general networking issue, but not sure why it can’t comm with the proxy.


Problem adding new node to cluster (certificate problem)
(Matt Bentley) #16

@ashbyj - Could you try to do a curl command to see if you can manually communicate to the ucp-proxy container? You’ll want to use the certs that are found in the ucp-node-certs volume so you may need to do a docker volume inspect ucp-node-certs to get the Mountpoint location which should default to /var/lib/docker/volumes/ucp-node-certs/_data if you’re using the default graph directory.

cd /var/lib/docker/volumes/ucp-node-certs/_data
curl -v --cacert ca.pem --cert cert.pem --key key.pem https://docker.example.com:12376/info

You should get the JSON output from the /info API endpoint if it is working as expected along with the debug output from curl. If you don’t it, it would be interesting to see what might be happening.

If that does work, you may also want to test from inside a container like so:

docker run -it --rm -v ucp-node-certs:/certs mbentley/curl -v --cacert /certs/ca.pem --cert /certs/cert.pem --key /certs/key.pem https://docker.example.com:12376/info

This will mount the certs into the container and use curl to try to talk to the ucp-proxy container. Here is my output as an example:

# docker run -it --rm -v ucp-node-certs:/certs mbentley/curl -v --cacert /certs/ca.pem --cert /certs/cert.pem --key /certs/key.pem https://localdev:12376/info
*   Trying 192.168.56.4...
* Connected to localdev (192.168.56.4) port 12376 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /certs/ca.pem
  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS handshake, CERT verify (15):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*        subject: OU=ucp; CN=swarm
*        start date: Mar 11 19:31:00 2016 GMT
*        expire date: Mar  9 19:31:00 2026 GMT
*        subjectAltName: localdev matched
*        issuer: CN=UCP Cluster Root CA
*        SSL certificate verify ok.
> GET /info HTTP/1.1
> Host: localdev:12376
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Server: Docker/1.10.3 (linux)
< Date: Tue, 15 Mar 2016 14:36:40 GMT
< Content-Length: 1455
<
{"ID":"2GH4:KUEP:Q5HH:I6T7:KZM6:2AOC:MA5S:QSVV:FJ72:IPR5:7ABX:4MR2","Containers":8,"ContainersRunning":8,"ContainersPaused":0,"ContainersStopped":0,"Images":460,"Driver":"aufs","DriverStatus":[["Root Dir","/var/lib/docker/aufs"],["Backing Filesystem","extfs"],["Dirs","624"],["Dirperm1 Supported","true"]],"SystemStatus":null,"Plugins":{"Volume":["local"],"Network":["null","host","bridge"],"Authorization":null},"MemoryLimit":true,"SwapLimit":true,"CpuCfsPeriod":false,"CpuCfsQuota":false,"CPUShares":true,"CPUSet":true,"IPv4Forwarding":true,"BridgeNfIptables":true,"BridgeNfIp6tables":true,"Debug":false,"NFd":84,"OomKillDisable":true,"NGoroutines":111,"SystemTime":"2016-03-15T10:36:40.185472944-04:00","ExecutionDriver":"native-0.2","LoggingDriver":"json-file","NEventsListener":1,"KernelVersion":"3.16.0-4-amd64","OperatingSystem":"Debian GNU/Linux stretch/sid","OSType":"linux","Architecture":"x86_64","IndexServerAddress":"https://index.docker.io/v1/","RegistryConfig":{"InsecureRegistryCIDRs":["127.0.0.0/8"],"IndexConfigs":{"docker.io":{"Name":"docker.io","Mirrors":null,"Secure":true,"Official":true}},"Mirrors":null},"InitSha1":"cd6756fea351239433c37a792d4626b75eb96d50","InitPath":"/usr/lib/docker/dockerinit","NCPU":1,"MemTotal":2108043264,"DockerRootDir":"/var/lib/docker","HttpProxy":"","HttpsProxy":"","NoProxy":"","Name":"localdev","Labels":null,"ExperimentalBuild":false,"ServerVersion":"1.10.3","ClusterStore":"","ClusterAdvertise":""}
* Connection #0 to host localdev left intact

Thanks,
Matt


UCP installation Error
(Ashbyj) #17

Thanks Matt. The curl command from the host was successful:

root@docker:/var/lib/docker/volumes/ucp-node-certs/_data> curl -v --cacert ./ca.pem --cert ./cert.pem --key ./key.pem https://docker.example.com:12376/info

and it returned the curl debug output as well as the json output.

However, curl does not work from inside a container

root@docker:/var/lib/docker/volumes/ucp-node-certs/_data> docker run -it --rm -v ucp-node-certs:/certs mbentley/curl -v --cacert /certs/ca.pem --cert /certs/cert.pem --key /certs/key.pem https://docker.example.com:12376/info
Unable to find image 'mbentley/curl:latest' locally
latest: Pulling from mbentley/curl
ee54741ab35b: Pull complete
6d240d52d74a: Pull complete
Digest: sha256:99e370871c9f81a7f19a5aecc4db83c96514f465628818f086dbf132b54437d8
Status: Downloaded newer image for mbentley/curl:latest
*   Trying 172.22.4.4...
* connect to 172.22.4.4 port 12376 failed: Host is unreachable
* Failed to connect to docker.example.com port 12376: Host is unreachable
* Closing connection 0
curl: (7) Failed to connect to docker.example.com port 12376: Host is unreachable

(Matt Bentley) #18

Is there a proxy server in your environment by chance where the proxy server info may be required to talk to a host? Any daemon arguments you’ve specified that may change how Docker networking is set up my default (like preventing IP forwarding from being enabled by Docker or by configuration management, etc)?

As a test, would you be able to see if the same problem occurs when you specify --host-address as the host’s IP address instead of hostname?

Thanks,
Matt


(Ashbyj) #19

I had to explicitly set this kernel param (set via puppet if that matters):

$ sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

as its disabled by default on CentOS 7. This github issue is where I landed for that particular problem of containers not getting outbound internet access.

No special daemon options. Here is the systemd init settings for docker:

$ cat /usr/lib/systemd/system/docker.service
[Unit]
Description=Docker Application Container Engine
Documentation=https://docs.docker.com
After=network.target docker.socket
Requires=docker.socket

[Service]
Type=notify
ExecStart=/usr/bin/docker daemon -H fd://
MountFlags=slave
LimitNOFILE=1048576
LimitNPROC=1048576
LimitCORE=infinity
TimeoutStartSec=0

[Install]
WantedBy=multi-user.target

I did try with ExecStart=/usr/bin/docker daemon -H tcp://0.0.0.0:2375 -H unix:///var/run/docker.sock --exec-opt native.cgroupdriver=cgroupfs as some CentOS 7 users have needed, but I reverted it back to the default per above. Neither work in regards to this issue.


(Ashbyj) #20

To answer your other question, there are no proxy servers in our environment here. And I’m doing all of this on the same VM.