Dockerize Mapr Cluster on a single server

Is it possible to run multiple containers in a single server and run a cluster?

Yes, its possible. I have run multiple clusters for testing purpose on multiple containers.

You can find some of my sample works here. Its mostly based on Cloudera.

I see so it is possible. Can you advice me on my analogy on making this possible? Here are the steps im playing on my mind.

First I need to download a base image (centos) then run it multiple times (3 containers)
Setup a network for those 3 to communicate
And that would end up as a cluster.

You are almost right.

From my experience this is what I do. I use ansible for creating Docker network and containers.

- name: Create Docker Network
        name: "{{ network_name }}"

- name: Create Hadoop Docker Cluster
    image: "{{ docker_image }}"
    name: node-{{ item }}
    hostname: node-{{ item }}
    network_mode: bridge
      - name: "{{ network_name }}"
    purge_networks: true
    command: /bin/sleep infinity
    privileged: true
  with_sequence: count=6
  • Step 1 - Create a Docker user defined network
  • Step 2 - Spin up multiple containers attached to that network (this will help in inbuilt DNS resolution)

One points to note in step 2 is keep your container name and hostname same, and short name. Because docker internal DNS will use container_name.network_name as the fqdn. If you follow this the DNS resolution inside your containers will be perfect (which is very much required especially in a hadoop env).

Let me know how you go with this setup.

1 Like

Ok sure! , technically you are correct on steop 2 note with the name and hostname the same. Going to see this ansible for creating docker network and container never heard of it before

You could use Docker Compose for the same. I prefer Ansible just because its something which I use for automating infrastructure.