Container to container traffic blocked

lasitus · December 19, 2019, 10:55pm

I have an issue where docker containers launched from a docker-compose file can’t communicate with each other. I have tried no specific network (docker-compose auto generated), a network specified in the docker-compose file, and an external network created manually on the command line and specified as the default network for the compose file. I have removed all of my personal images and have been using a network test image and ssh servers. I have been testing by running nmap on the network test image. In all cases, I can ping the other computer using the name. I can also ping using the IP address, but connecting to the port in question and testing the port using nmap and curl, I can’t connect from within a container. On the host, I can port scan using the ip address.

I have performed a full system prune, and even removed docker/docker-compose, deleted /var/lib/docker, and reinstalled and still have the issue.

I tried the same compose file on another system with the same base OS and versions of software, and it does not show the issue. I tried purging the firewall and compared both machine’s configuration and iptables rules. They are identical… though there was an error originally in the config file on the system that doesn’t work (I have since corrected it and performed a full reinstall though)

system info:

Client:
 Debug Mode: false

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 2
 Server Version: 19.03.5-ce
 Storage Driver: btrfs
  Build Version: Btrfs v5.3.1 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: d50db0a42053864a270f648048f9a8b4f24eced3.m
 runc version: d736ef14f0288d6993a1845745d6756cfc9ddd5a
 init version: fec3683
 Security Options:
  seccomp
   Profile: default
 Kernel Version: 5.3.15-1-MANJARO
 Operating System: Manjaro Linux
 OSType: linux
 Architecture: x86_64
 CPUs: 24
 Total Memory: 31.31GiB
 Name: adampc
 ID: CAYT:SOPG:EGJ6:7VWB:CIKD:I72K:S2DN:PCY4:R2W3:LIMP:THQQ:PBSV
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

docker-compose.yml:

version: '2.3'

services:
  sshserver:
    image: linuxserver/openssh-server
    #hostname: sshserver
  sshserver2:
    image: linuxserver/openssh-server
    #hostname: sshserver2  
  nettest:
    image: praqma/network-multitool
    #hostname: nettest

cleanstart.sh

sudo systemctl restart docker
docker-compose up -d
docker exec -it networktest_nettest_1 bash -c 'nmap sshserver'

cleanstop.sh

docker-compose down
docker system prune -af
sudo systemctl stop docker
sudo rm -rf /var/lib/docker
sudo iptables -F
sudo iptables -X

Output on the machine that works:

Creating network "networktest_default" with the default driver
Pulling sshserver (linuxserver/openssh-server:)...
latest: Pulling from linuxserver/openssh-server
0fe9c6a00d3e: Pull complete
11b8fb9e70b1: Pull complete
dbb5082de684: Pull complete
31df4dd9c3f4: Pull complete
7846c7ca02ba: Pull complete
Digest: sha256:ef6b1005563090b13baa3a309a0025ad1b88eeee6c2e0afaee0c18c6404e4a9b
Status: Downloaded newer image for linuxserver/openssh-server:latest
Pulling nettest (praqma/network-multitool:)...
latest: Pulling from praqma/network-multitool
050382585609: Pull complete
d1e342a34d6b: Pull complete
079d5234adcd: Pull complete
1deecd267281: Pull complete
d9d9efb6576a: Pull complete
a84544204238: Pull complete
a67bb2f07ab1: Pull complete
Digest: sha256:69f3947ff89b80abd8bfad0d12047fec820f14f8184a9e6aa27487fa6df6a79c
Status: Downloaded newer image for praqma/network-multitool:latest
Creating networktest_nettest_1    ... done
Creating networktest_sshserver2_1 ... done
Creating networktest_sshserver_1  ... done
Starting Nmap 7.70 ( https://nmap.org ) at 2019-12-20 14:06 UTC
Nmap scan report for sshserver (172.18.0.4)
Host is up (0.000019s latency).
rDNS record for 172.18.0.4: networktest_sshserver_1.networktest_default
Not shown: 999 closed ports
PORT     STATE SERVICE
2222/tcp open  EtherNetIP-1
MAC Address: 02:42:AC:12:00:04 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 1.69 seconds

Output on the machine that doesn’t work:

Creating network "networktest_default" with the default driver
Pulling sshserver (linuxserver/openssh-server:)...
latest: Pulling from linuxserver/openssh-server
0fe9c6a00d3e: Pull complete
11b8fb9e70b1: Pull complete
dbb5082de684: Pull complete
31df4dd9c3f4: Pull complete
7846c7ca02ba: Pull complete
Digest: sha256:ef6b1005563090b13baa3a309a0025ad1b88eeee6c2e0afaee0c18c6404e4a9b
Status: Downloaded newer image for linuxserver/openssh-server:latest
Pulling nettest (praqma/network-multitool:)...
latest: Pulling from praqma/network-multitool
050382585609: Pull complete
d1e342a34d6b: Pull complete
079d5234adcd: Pull complete
1deecd267281: Pull complete
d9d9efb6576a: Pull complete
a84544204238: Pull complete
a67bb2f07ab1: Pull complete
Digest: sha256:69f3947ff89b80abd8bfad0d12047fec820f14f8184a9e6aa27487fa6df6a79c
Status: Downloaded newer image for praqma/network-multitool:latest
Creating networktest_sshserver_1  ... done
Creating networktest_sshserver2_1 ... done
Creating networktest_nettest_1    ... done
Starting Nmap 7.70 ( https://nmap.org ) at 2019-12-20 14:07 UTC
Nmap scan report for sshserver (172.18.0.2)
Host is up (0.000026s latency).
rDNS record for 172.18.0.2: networktest_sshserver_1.networktest_default
All 1000 scanned ports on sshserver (172.18.0.2) are filtered
MAC Address: 02:42:AC:12:00:02 (Unknown)

Nmap done: 1 IP address (1 host up) scanned in 7.20 seconds

Firewall rules on non-working machine

sudo iptables -L
[sudo] password for adam: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

firewall rules on working machine:

[sudo] password for adam: 
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy DROP)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere

lasitus · January 3, 2020, 7:38pm

Anything else I can purge when reinstalling docker?

Topic		Replies	Views
Docker Containers Outbound Traffic doesn't connect General docker	1	670	December 10, 2022
Communication between Networks with different drivers General docker , docker-compose	2	638	June 3, 2023
Help with Docker networking General docker	5	1923	May 27, 2020
Docker connection refused between two containers General	14	103370	November 6, 2024
Communication error between host and docker containers in user defined bridged network General docker , docker-compose	0	810	March 8, 2022

Container to container traffic blocked

Related topics