Docker Community Forums

Share and learn in the Docker community.

Docker Containers: Error Unable to establish connection

I have odd issue on RHEL server running 4 docker containers. 2 containers that connect to Cluster A started failing to connect. While the other 2 connect to the same Cluster A with out issue.

These two containers were previously working, and I am unable to determine why since all four connect to the same cluster. I am also able to use CURL to verify the connectivity to Cluster A is there.

My suspicion is maybe this is a issue with the two failing containers getting out due to IPTables maybe?

Thoughts?

**###Containers connecting to cluster "Cluster A" not working as intended###**
CONTAINER ID   IMAGE                                COMMAND                  CREATED      STATUS         PORTS                                                                     NAMES
65825103ed0d   rubrikinc/rubrik-prometheus-client   "/main"                  3 days ago   Up 3 days      0.0.0.0:9480->8080/tcp, :::9480->8080/tcp                                 jovial_jepsen
e86307b0244f   rubrikinc/rubrik-prometheus-client   "/main"                  3 days ago   Up 5 seconds   0.0.0.0:9479->8080/tcp, :::9479->8080/tcp                                 festive_matsumoto

**###Docker Logs Show Failure to connect###**
021/04/30 05:27:56 Cluster name: Cluster A
2021/04/30 05:27:56 Starting on HTTP port 8080
2021/04/30 05:28:56 Error from jobs.GetMssqlFailedJobs: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)
2021/04/30 05:34:52 Error from stats.GetNodeStats: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)
2021/04/30 05:35:04 Error from jobs.GetMssqlFailedJobs: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)

**###Containers connecting to cluster Cluster A working as intended###**
06d511124e50   claranet/rubrik-exporter             "/usr/local/bin/rubr…"   5 days ago   Up 3 days      0.0.0.0:9478->9477/tcp, :::9478->9477/tcp                                 compassionate_hodgkin
20de8ea6c6e7   claranet/rubrik-exporter             "/usr/local/bin/rubr…"   5 days ago   Up 3 days      0.0.0.0:9477->9477/tcp, :::9477->9477/tcp                                 ecstatic_mahavira

**####Docker logs show success###**
time="2021-05-03T14:44:10Z" level=info msg="Requested action: /api/internal/node/******************/stats" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/stats/system_storage" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/archive/location" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/stats/data_location/usage" file=rubrik_api.go line=43
time="2021-05-03T14:44:12Z" level=info msg="Requested action: /api/internal/stats/physical_ingest/time_series" file=rubrik_api.go line=43
time="2021-05-03T14:46:23Z" level=info msg="Requested action: /api/internal/stats/per_vm_storage" file=rubrik_api.go line=43

**###Curl is able to connect from the host running docker, to "Cluster A" with out issue.###**
# curl -k -u https://xxxxxxxxxxxxx/api/internal/stats/system_storage
{"total":330629660151808,"used":263149320282112,"available":67480339869696,"snapshot":256996418231787,"liveMount":0,"pendingSnapshot":0,"cdp":0,"miscellaneous":6152902050325,"lastUpdateTime":"2021-04-30T21:35:58.443Z"}

# curl -k -u https://xxxxxxxxxxxxx/api/internal/stats/system_storage
{"total":360686901985280,"used":307824132628480,"available":52862769356800,"snapshot":300965018830322,"liveMount":0,"pendingSnapshot":0,"cdp":0,"miscellaneous":6859113798158,"lastUpdateTime":"2021-04-30T21:39:49.891Z"}



**###Info about the setup###**
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

]# docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:43:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:callwaveiam
ACCEPT     tcp  --  anywhere             172.17.0.6           tcp dpt:9477
ACCEPT     tcp  --  anywhere             172.17.0.4           tcp dpt:9477
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:webcache
ACCEPT     tcp  --  anywhere             172.17.0.5           tcp dpt:webcache

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

by default, docker-compose create a dedicated network for the services that you are running. When you run the docker instance using docker run and you do not specify in which network should be run, it will run on the default network.
This could be the problem and you can solve it by running the docker instance that do not use compose on the docker-compose network. To obtain the name of the network you can run

docker inspect <compose_container_id> | grep -i “NetworkMode”
Once you get the name of the network, you can run the container, connect to the network and link with the services of the compose instances.

docker run -it --network <network_name> --link <compose_container_name> <docker_image>
On the other hand, I don’t know why the error that you pasted is showing that you are trying to connect to localhost.

HTTPConnectionPool(host=‘localhost’, port=5000)
In case that you want to connect to a host mapped port, you have to specify the host ip that is connected to the container network.

I hope you find this useful.

Thanks for the info, ill give this a try. At this point every bit helps.