Stack and Hadoop

Docker Engine Swarm
1

I’m having some issues trying to deploy a simple hadoop cluster using docker stack deploy

OS: Raspian
Docker version: 17.05.0-ce

my docker-compose.yml file looks like:

version: '3.2'

services:

    hadoop-master:
        image: sumglobal/rpi-hadoop:2.7.2
        ports:
            - "50070:50070"
            - "8088:8088"
            - "8031:8031"
        command: master

    worker-1:
        image: sumglobal/rpi-hadoop:2.7.2
        command: worker
        ports:
            - "50075"
            - "8142:8042"
            - "19888:19888"

When i deploy with docker stack deploy -c docker-compose.yml hadoop-cluster, it looks like it all comes up clean but the worker node shows the following errors:

17/06/30 13:57:50 INFO ipc.Client: Retrying connect to server: hadoop-master/10.0.0.2:8031. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/30 13:57:51 INFO ipc.Client: Retrying connect to server: hadoop-master/10.0.0.2:8031. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
17/06/30 13:57:52 INFO ipc.Client: Retrying connect to server: hadoop-master/10.0.0.2:8031. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)

which is continually repeated. when attaching to the worker node, i can ping that ip address.

attaching to the master node shows:

root@c500fa3d35d3:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
72: eth0@if73: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 02:42:0a:00:00:03 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.3/24 scope global eth0
       valid_lft forever preferred_lft forever
    inet 10.0.0.2/32 scope global eth0
       valid_lft forever preferred_lft forever
74: eth1@if75: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:ac:12:00:03 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.3/16 scope global eth1
       valid_lft forever preferred_lft forever
76: eth2@if77: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default
    link/ether 02:42:0a:ff:00:07 brd ff:ff:ff:ff:ff:ff
    inet 10.255.0.7/16 scope global eth2
       valid_lft forever preferred_lft forever
    inet 10.255.0.2/32 scope global eth2
       valid_lft forever preferred_lft forever

which lists 10.0.0.2 but also 10.0.0.3 (which admittedly also confuses me). the logs on the namenode seem to indicate the port is being listened on:

17/06/30 13:57:35 INFO ipc.Server: Starting Socket Reader #1 for port 8031

I appreciate any help.

Thanks,
Charlie

2

Hi,
I am also facing the same problem.