4 node Cloudera hadoop cluster on docker containers

General Discussions
1

I am setting up 4 nodes cloudera hadoop cluster on docker containers using cloudera manager, I have installed cloudera manager successfully but at the time of adding nodes to cluster I am facing authentication problem. Authentication can be done by providing private key or by providing password, I m not getting what password should I provide as docker containers do not contain root passwords, also m not aware about the process of providing private key.
It will be really appretiable if any one could help me with this problem.

2

Hi,

Hi, which baseimage are you using for setting up the worker nodes? Do you have SSH installed and enabled in the worker node docker images? You will have to setup SSH and configure password for your container.

Regards

1 Like
3

Hi,
I am using ubuntu 14.04 as base image and cloudera manager is also installed in a container using ubuntu 14.04 as a base image. I have also installed ssh in all my containers and generated keys as well, m also able to ssh in all the containers from the one in which cloudera manager is installed. Pls help me in configuring password for my container.
And let me know what option should i use while authentication ie : using password or by providing private key and pass phrase.

Regards

4

Hi,

I think it will be good if you can use ssh keys with/without a passphrase, this will ease in container automation. In your case you can use a single ssh-keypair and distribute the public key to all worker nodes authorized_keys and use the private key in Cloudera Manager to connect and manage worker nodes.

It would be easy to collaborate if you can share the Dockerfile you are using to build the entire system. Also there are many other repos and project that already has build hadoop containers which you can reuse. In case you want to anyway build from scratch I am more than happy to support.

Regards.

5

Thankyou for the pointers.

As i am new to docker I m doing all the operations manually I am not aware if the docker file gets generated automatically. And yes I actually want to build the containers from scratch . I already used the process of authorized keys you mentioned above i.e. I made ssh connection from cloudera container to all the other containers (worker nodes) , also tried using password less ssh login. I used to upload the private key of cloudera container when asked for authentication but the browser actually gets hanged . Am I going right if yes are there any solns to my encountered problem.

Regards.

6

Can any one provide pointers for above problem?

7

Hi,

In your previous question you asked whether you are going right. I cannot say that you are wrong, but I can definitely say that its not the right way with Docker. Please spend some time getting the stuffs created using Dockerfile, so that you can distribute it in case if someone want to replicate what you are doing to help you.

Anyway I will try to build a sample cluster to help you. Please let me know which version of CDH are you using?

Regards

8

I am using cdh 5.1.2

9

Authentication with ssh private key ( No provider available)

can u pls help me with this.

10

D’OH, I realized suddeny that this question is a bit old
Hello everybody.
rshri, I suggest you to:

  • build a public/private RSA keypair (as it seems to be more widely accepted than the DSA alternative) for the “root” user of one of your Docker hosts as described for example here https://www.hostinger.com/tutorials/ssh/how-to-set-up-ssh-keys
  • use the ssh-copy-id command pointing to root@localhost in order to generate a properly formatted authorized_keys file inside your /root/.ssh/ folder
  • create a Dockerfile using RUN and COPY istructions: plenty of examples/tutorials, try first with the official one https://docs.docker.com/engine/reference/builder/

something like this should be run on your Docker host
ssh-keygen -t rsa -P '' -N '' -q -f ./id_rsa
ssh-copy-id -i ./id_rsa localhost

This will leave you with three files
id_rsa
id_rsa.pub
~/.ssh/authorized_keys

You should copy these files on the same directory of your Dockerfile which in turn should contain these lines (I will verify this evening)

RUN mkdir /root/.ssh && chmod 600 /root/.ssh
COPY ./id_rs* ./auth* /root/.ssh/
RUN chmod 600 /root/.ssh/*
RUN /usr/sbin/sshd

this should be sufficient to allow node-to-node SSH communications for the root user