Docker Community Forums

Share and learn in the Docker community.

Docker Container Connection Error

I have odd issue on RHEL server running 4 docker containers. 2 containers that connect to Cluster A started failing to connect. While the other two that connect to the same Cluster A are still working as intended.

These two containers were previously working, and I am unable to determine why since all four connect to the same cluster. I am also able to use CURL to verify the connectivity to Cluster A is there.

My suspicion is maybe this is a issue with the two failing containers getting out due to IPTables maybe?

Thoughts?

**###Containers connecting to cluster Cluster A not working as intended###**
CONTAINER ID   IMAGE                                COMMAND                  CREATED      STATUS         PORTS                                                                     NAMES
65825103ed0d   rubrikinc/rubrik-prometheus-client   "/main"                  3 days ago   Up 3 days      0.0.0.0:9480->8080/tcp, :::9480->8080/tcp                                 jovial_jepsen
e86307b0244f   rubrikinc/rubrik-prometheus-client   "/main"                  3 days ago   Up 5 seconds   0.0.0.0:9479->8080/tcp, :::9479->8080/tcp                                 festive_matsumoto

**###Docker Logs Show Failure to connect###**
021/04/30 05:27:56 Cluster name: Cluster A
2021/04/30 05:27:56 Starting on HTTP port 8080
2021/04/30 05:28:56 Error from jobs.GetMssqlFailedJobs: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)
2021/04/30 05:34:52 Error from stats.GetNodeStats: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)
2021/04/30 05:35:04 Error from jobs.GetMssqlFailedJobs: %!(EXTRA *errors.errorString=Unable to establish a connection to the Rubrik cluster)

**###Containers connecting to cluster Cluster A working as intended###**
06d511124e50   claranet/rubrik-exporter             "/usr/local/bin/rubr…"   5 days ago   Up 3 days      0.0.0.0:9478->9477/tcp, :::9478->9477/tcp                                 compassionate_hodgkin
20de8ea6c6e7   claranet/rubrik-exporter             "/usr/local/bin/rubr…"   5 days ago   Up 3 days      0.0.0.0:9477->9477/tcp, :::9477->9477/tcp                                 ecstatic_mahavira

**####Docker logs show success###**
time="2021-05-03T14:44:10Z" level=info msg="Requested action: /api/internal/node/******************/stats" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/stats/system_storage" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/archive/location" file=rubrik_api.go line=43
time="2021-05-03T14:44:11Z" level=info msg="Requested action: /api/internal/stats/data_location/usage" file=rubrik_api.go line=43
time="2021-05-03T14:44:12Z" level=info msg="Requested action: /api/internal/stats/physical_ingest/time_series" file=rubrik_api.go line=43
time="2021-05-03T14:46:23Z" level=info msg="Requested action: /api/internal/stats/per_vm_storage" file=rubrik_api.go line=43

**###Curl is able to connect from the host running docker, to Cluster A with out issue.###**
# curl -k -u /api/internal/stats/system_storage
{"total":330629660151808,"used":263149320282112,"available":67480339869696,"snapshot":256996418231787,"liveMount":0,"pendingSnapshot":0,"cdp":0,"miscellaneous":6152902050325,"lastUpdateTime":"2021-04-30T21:35:58.443Z"}

# curl -k -u api/internal/stats/system_storage
{"total":360686901985280,"used":307824132628480,"available":52862769356800,"snapshot":300965018830322,"liveMount":0,"pendingSnapshot":0,"cdp":0,"miscellaneous":6859113798158,"lastUpdateTime":"2021-04-30T21:39:49.891Z"}



**###Info about the setup###**
# cat /etc/redhat-release 
Red Hat Enterprise Linux Server release 7.9 (Maipo)

]# docker version
Client: Docker Engine - Community
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.13.15
 Git commit:        370c289
 Built:             Fri Apr  9 22:45:33 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server: Docker Engine - Community
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.13.15
  Git commit:       8728dd2
  Built:            Fri Apr  9 22:43:57 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.4.4
  GitCommit:        05f951a3781f4f2c1911b05e61c160e9c30eaa8e
 runc:
  Version:          1.0.0-rc93
  GitCommit:        12644e614e25b05da6fd08a38ffa0cfe1903fdec
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0


# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
REJECT     all  --  anywhere             anywhere             reject-with icmp-host-prohibited

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (1 references)
target     prot opt source               destination         
ACCEPT     tcp  --  anywhere             172.17.0.2           tcp dpt:callwaveiam
ACCEPT     tcp  --  anywhere             172.17.0.6           tcp dpt:9477
ACCEPT     tcp  --  anywhere             172.17.0.4           tcp dpt:9477
ACCEPT     tcp  --  anywhere             172.17.0.3           tcp dpt:webcache
ACCEPT     tcp  --  anywhere             172.17.0.5           tcp dpt:webcache

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

Posted in wrong place. Moderator please delete.