Docker Community Forums

Share and learn in the Docker community.

UCP has trouble to connect newly added node (worker)

(Binyang2) #1

Added new node to docker swarm, it looks good.

docker swarm join --token SWMTKN-1-1omic2el5jd3g98lm9x77o0px6ymauv3brjomfist1o2miga1x-c925hpkai0wm8pg5a01k2v9rv

This node joined a swarm as a worker.

  1. From UCP machine run: docker node list
    I can see the new node STATUS is Ready, and AVAILABILITY is Active

  2. However, at UCP Web UI, the new node Status is Down with the following message:
    Unable to allocate the required ports for UCP, please visit for more information.

How does this can be fixed? Does docker/ucp-agent proxy container have to be manually installed and run on the new node?


(Wsongdocker) #2

It sounds like your new node doesn’t have port 12376 open. If the proxy container can’t start because it can’t acquire port 12376, the new node will still show up in docker node ls, but UCP will mark it as down until you open up port 12376 on that machine.

(Binyang2) #3

Thanks for the reply.

I have firewalld & iptables services stopped, and port 12376 is open.

Current there is no any container running on the new node. Do I need to manually put ucp-agent container on the new node?


(Wsongdocker) #4

Take a look and see if there are any stopped containers on the new node with docker ps -a. If there are, take a look at their logs. If not, the problem may be that the ucp-agent image wasn’t able to get pulled on your new node. Make sure you run docker pull docker/ucp-agent:<version> where <version> is the version of UCP you’re running (e.g. 2.0.2).

Also, you can run docker service ps ucp-agent on one of your existing manager nodes to see the list of ucp-agent tasks in the cluster. This may contain more info about a failed task on your new node.

(Binyang2) #5

How does ucp-agent container get on the new node at the first place? Does it push over by swarm manager when I run “docker swarm join” ?

(Wsongdocker) #6

That’s correct, although note that here the “Swarm manager” refers to the Swarm leader you can see when you run docker node ls.

(Binyang2) #7

Thanks for the information.

In my case, after “docker swarm join”, no docker image was created at new node.

Where should I look for debug error message?

(Wsongdocker) #8

Check out docker service ps ucp-agent and see if there are any interesting error messages. Also make sure that you have the ucp-agent image on the new node.

(Binyang2) #9

You mean I may need to manually pull ucp-agent image to the new node?

(Wsongdocker) #10

It’s possible. The error message in docker service ps should have more information.

(Binyang2) #11

Here is what I have, the new node is docker-node3, it does not show up at “docker service ps ucp-agent” output

[root@docker-ucp ~]# docker node list
6neg6eapl8qkakh426p275uhf * docker-ucp Ready Active Leader
hntvi6edvjjm8re2zdhuglxts docker-dtr Ready Active
jb2t1irplx8acpcm5rggac0o4 docker-node3 Ready Active
mbu5hmg9dyoiq1alg7afen2fs docker-dtr01 Ready Active

[root@docker-ucp ~]# docker service ps ucp-agent
ucp-agent.0.rjofcjrvrjq7 docker/ucp-agent:2.0.1 docker-dtr01 Running Running about an hour ago
_ ucp-agent.0.ntw66klmm1oy docker/ucp-agent:2.0.1 docker-dtr Running Running 22 hours ago
_ ucp-agent.0.m7uti146imr5 docker/ucp-agent:2.0.1 docker-ucp Running Running 5 weeks ago

(Wsongdocker) #12

It looks like your Swarm cluster is having trouble scheduling the ucp-agent task on the new node. Try looking at the daemon logs on the new node and on the Swarm leader (i.e. the docker-ucp node) to see if there’s anything interesting. You can find the logs here:

(Binyang2) #13

Finally, the new node seems happy in the swarm…

I found the error message in stopped ucp-agent container by “docker logs 959361852bd5”:

Error while checking for available ports: The following required ports are already in use on your host - 12376.

Reboot my host, it is good.

Thank you wsongdocker

(Berttejeda) #14

In my case, I was working with an air-gapped system, so the worker node could not communicate with the upstream docker registry ( Tell-tale sign was the error level=error msg="pulling image failed" error="Get in /var/log/messages. The solution was to gnab all UCP-related images on a less restricted system and upload to the worker:
shopt -s extglob;docker container run --rm -it --name ucp -v /var/run/docker.sock:/var/run/docker.sock docker/ucp:2.2.4 images --list | while read image;do docker pull "${image} && docker save -o "${image//@(\/|:)/_}.tar" && scp "${image//@(\/|:)/_}.tar" myusername@my.worker.node:/home/myusername && ssh myusername@my.worker.node "sudo docker load -i ${image//@(\/|:)/_}.tar";done

I haven’t tested the above, but you get the gist.