I have successfully established communication with Hive through port 10000. However, I would like to expose port 8020 on the Namenode for Hadoop.
When I enter the Namenode’s bash and run the netstat command, I see the following output:
$ docker exec -it namenode /bin/bash
root@aa6ef06546d3:/# netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 localhost:39240 localhost:50070 TIME_WAIT
tcp 0 0 aa6ef06546d3:8020 datanode.hive_def:43322 ESTABLISHED
tcp 0 0 localhost:39296 localhost:50070 TIME_WAIT
I would like to bind the established address (aa6ef06546d3:8020) to localhost:8020 so that I can connect it to Talend.
I attempted to add the following line to the docker-compose file under “ports”: “8020:namenode:8020”, but I received the error message “1 error(s) decoding: * error decoding ‘ports’: Invalid IP address: 8020”.
Do you have any suggestions on how to bind this established address to localhost while maintaining the connection to Hive?
The address aa6ef06546d3:8020 is already accessible within the container itself (aa6ef06546d3) and is exposed to another container (datanode) within the same docker-compose setup.
However, I also require the address aa6ef06546d3:8020 to be accessible from localhost (the host machine) so that I can connect the namenode to Talend running on the host.
I tried using “8020:8020” in the ports configuration. However, after running the docker ps command, port 8020 is exposed to localhost, but it still doesn’t map the address aa6ef06546d3:8020 to the localhost.
If the internal port can not be exposed because it’s listening on a dedicated internal host/IP, then you need to configure namenode listen internally on all addresses.
I still don’t understand it. The netstat output you shared shows that the process is listening on the container’s ip address, not on localhost and the fact that you can access it from another container proves that it is accssible from other IPs. The answer @bluepuma gave you was correct based on what you shared. That forwards all requests from localhost:8020 to the container’s ip address on which the process is listening.
Please, show your compose file with your portmapping, because I suspect you just addedd the mapping to a wrong place in the file. Or you were not able to explain what you actually want to achieve. For example this doesn’t make sense to me:
You have a process listening on a port in the container. You can forward requests from your host to the container but you don’t need to forward it to localhost again. The process already got the request. Unless you meant the opposite and you want to forward a port inside the container to the host outside the container.
The part of the above sentence in the parenthesis and outside are contradict each other. It’s either you make a container port acesible from the host outside the container or let a process (datanode) inside the container to access a port on the host.
If port 8020 is on which the datanode is listening than the port has nothing to do with how it can access ports on the host. You can use host.docker.internal hostname in Docker Desktop to access something on the physical host outside the container or do something similar without Docker Desktop on Linux:
but rather connect Talend to namenode. That should work so something else is going on but the port and IP seems right
And it was done as the netstat command showed
It means the datanode container could connect to namenode using the containers ip address. It isn’t just listening on the container’s ip probably, but this netstat info shows only the established connection that happened on the container’s IP to which the hostname belongs.
That’s why I asked for a compose file.
@hagarusama Without seeing a compose file, Dockerfile or knowing what image you are using exactly and how you try to connect to the port exactly and without any error message there is nothing more we can do. We are just constantly guessing.
Please bear with me as it may take some time to write the reply. I appreciate your patience, and I’ll get back to you as soon as possible with the information you need.
Without seeing a compose file, Dockerfile or knowing what image you are using exactly and how you try to connect to the port exactly and without any error message there is nothing more we can do. We are just constantly guessing.
I followed the tutorial exactly as it was, without making any modifications. Here are the docker-compose.yml and Hadoop-Hive files that I used.