Share and learn in the Docker community.
Was expecting that we could ssh on worker nodes.
Using ssh with docker as the user, the 22 port connection is refused even if the SG is configured to allow it.
Hi there!
sshd does non run on the worker nodes (nor does it run directly on the managers). I’d love to learn your use case for requiring SSH access to the worker nodes, can you provide more details?
Michael
Hi Michael,
So long story short we are trying to spin up some microservices that are in the single private repo we have: hkmconsultingllc/ep… and scale them. We have a single manager node and 3 worker nodes spun up using the Cloud Formation template.
On the manager node we logged in and then pulled the image. We did this by accessing the manager node directly using the public IP from EC2 vs. the loadbalancer. This is just a note for you as we also have done it through the ELB. We used docker login to get to the private repo.
Once the image was pulled onto the manager we then used the docker deploy command below to create a service.
docker deploy
Once this was done we tried to scale the service with the following command. This did not show the replica being created.
docker service scale atru_consumerapp=3
All we see when we do docker service ls is 0/3 for the docker containers across the board vs. 3/3.
From a usability perspective we were hoping to use the worker node login to see what is going on using commands like docker ps -a, docker logs etc.
regarding the docker deploy command, for some reason the community editor removed that I had specified a dab file as the final parameter.
If you can also suggest any mechanism we should be using to debug what is going on in the service and why replicas are not spinning up.
We have resolved some of these issues but also identified a problem with private repos as noted in the new post, Use of Docker Private Repo doesn't work properly with Docker for AWS.
It would be helpful to understand how to debug issues on the worker nodes if we do not have access or special logging features.
Thanks for posting more details, and sorry about the slow response. I’ve replied to your other post about deploying services based on images in private repos.
We’re keenly aware that debugging facilities in Docker for Azure are very spartan. One thing that you can do right now is to run docker -H <node-host-name> logs foo-container using the hostnames shown in docker node ls. We’re looking at locking down the swarm more, so this will likely stop working in a future version.
We’re looking at logging to Cloudwatch. If you have other ideas or suggestions for how logging and debugging should work, we’d love to hear them.
We’re reluctant to allow direct SSH access to worker nodes because it would open up for customizations that don’t fit with the Docker for AWS operational model.
Hi Michael,
Thank you for taking time out to reply. We will try out the logs option you mentioned. Use of CloudWatch would make sense in leveraging the AWS platform for this so I would be all in for that. Otherwise we are looking at using LogStash to push the data out to ElasticSearch.
Will update this issue with what we run into and suggestions. My other cohort will probably have some suggestions.
thx
Chris
Hi guys,
I am currently in a similar situation and would like to docker exec into a task running on a worker node. Unfortunately, the suggested approach using docker -H <node-host-name> ... does not work for me - I receive Cannot connect to the Docker daemon.
Since this issue is quite old I wondered if any new approaches have been implemented for debugging?
You can SSH to the workers from the manager though.
Also, if you’re using a private registry, you need to login on the manager (docker login …) then deploy with the --with-registry-auth flag, This gives the workers the authority to pull from the registry that was logged on to on the manager.
e.g. ‘docker stack deploy --with-registy-auth …’ or 'docker service create --with-registry-auth … ’
Sorry if you’ve done this and it didn’t help, but I couldn’t see your deploy commands in your posts.
Drat, didn’t realise this thread had been necrobumped.
SSH from the manager to the worker works but it feels weird to have to copy my private key to the manager node in order to be able to access the worker. Even if I created a fresh key pair I would need to first ssh into the worker node once to register it…
You don’t, you use agent forwarding.
eval `ssh-agent`
ssh-add yourkeyfile
ssh -A your-manager-node
ssh workernode1
Or, if you’re staring from windows, add your key to pageant, then putty with agent forwarding turned on works the same way.
then you don’t copy keys anywhere.
If you don’t want to use the agent, you can use multi hop with ProxyCommand