Docker Community Forums

Share and learn in the Docker community.

Enabling communication between containers associated with different bridges

Hi,

I am new to Docker and I was trying out the following architecture:

  1. Container A: IP address: 172.17.0.2, Gateway: 172.17.0.1 (Associated with Default Bridge “docker0”)
  2. Container B: IP address: 172.20.0.2, Gateway: 172.20.0.1 (Associated with custom Bridge)

-> I was trying to get these 2 containers to communicate with each other. I tried doing this without changing any of the iptables rules and observed that this is not possible by default. In order to understand how the packets were being dropped, I investigated the iptables rules and observed that the rule under the DOCKER-ISOLATION-STAGE-2 chain was being hit (There was a DENY rule). Here is the output of the “iptables -vL” command:

ping from 172.17.0.2 to 172.20.0.2

$ iptables -vL
Chain INPUT (policy ACCEPT 10812 packets, 617K bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
14211 40M DOCKER-USER all – any any anywhere anywhere
14211 40M DOCKER-ISOLATION-STAGE-1 all – any any anywhere anywhere
6175 39M ACCEPT all – any br-e063dcaeaf08 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all – any br-e063dcaeaf08 anywhere anywhere
4018 289K ACCEPT all – br-e063dcaeaf08 !br-e063dcaeaf08 anywhere anywhere
0 0 ACCEPT all – br-e063dcaeaf08 br-e063dcaeaf08 anywhere anywhere
3763 36M ACCEPT all – any docker0 anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 DOCKER all – any docker0 anywhere anywhere
3255 229K ACCEPT all – docker0 !docker0 anywhere anywhere
0 0 ACCEPT all – docker0 docker0 anywhere anywhere

Chain OUTPUT (policy ACCEPT 13387 packets, 1031K bytes)
pkts bytes target prot opt in out source destination

Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
pkts bytes target prot opt in out source destination
5670 402K DOCKER-ISOLATION-STAGE-2 all – br-e063dcaeaf08 !br-e063dcaeaf08 anywhere anywhere
5406 410K DOCKER-ISOLATION-STAGE-2 all – docker0 !docker0 anywhere anywhere
17211 75M RETURN all – any any anywhere anywhere

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
pkts bytes target prot opt in out source destination
116 9744 ACCEPT all – any any anywhere anywhere
0 0 DROP all – any docker0 anywhere anywhere
7273 518K RETURN all – any any anywhere anywhere

Chain DOCKER-USER (1 references)
pkts bytes target prot opt in out source destination
21014 75M RETURN all – any any anywhere anywhere

-> Now, when I changed the DENY rule to ACCEPT, I was able to establish connectivity between the 2 containers. However, I am trying to understand the following:

Q. What is the complete flow of communication that is going on here? (Container -> Bridge -> Evaluation of iptables?)

According to this flow: http://www.adminsehow.com/2011/09/iptables-packet-traverse-map/, the PREROUTING chain under the NAT table should be evaluated first. However, when I look at the output of “iptables -vL -t nat”, I do not see the packets hitting the PREROUTING rule. Here is the output:

iptables -vL -t nat
Chain PREROUTING (policy ACCEPT 6986 packets, 314K bytes)
pkts bytes target prot opt in out source destination
16582 715K DOCKER all – any any anywhere anywhere ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 6460 packets, 271K bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 866 packets, 65459 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER all – any any anywhere !ip-127-0-0-0.ec2.internal/8 ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 866 packets, 65459 bytes)
pkts bytes target prot opt in out source destination
90 6105 MASQUERADE all – any !br-e063dcaeaf08 ip-172-20-0-0.ec2.internal/16 anywhere
75 5347 MASQUERADE all – any !docker0 ip-172-17-0-0.ec2.internal/16 anywhere

Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
0 0 RETURN all – br-e063dcaeaf08 any anywhere anywhere
0 0 RETURN all – docker0 any anywhere anywhere

-> I do see some packets for the PREROUTING chain, but when I keep running the same command and a simultaneous ping, I do not see the value incrementing here. Additionally, when I take a packet capture on the destination host (container with IP address 172.20.0.2), I see packets being sourced from the Custom Bridge’s IP address:

$ tcpdump -i eth0 icmp -nn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
16:16:01.988163 IP 172.20.0.1 > 172.20.0.2: ICMP echo request, id 302, seq 18, length 64
16:16:01.988180 IP 172.20.0.2 > 172.20.0.1: ICMP echo reply, id 302, seq 18, length 64
16:16:03.012162 IP 172.20.0.1 > 172.20.0.2: ICMP echo request, id 302, seq 19, length 64
16:16:03.012178 IP 172.20.0.2 > 172.20.0.1: ICMP echo reply, id 302, seq 19, length 64
16:16:04.036178 IP 172.20.0.1 > 172.20.0.2: ICMP echo request, id 302, seq 20, length 64
16:16:04.036193 IP 172.20.0.2 > 172.20.0.1: ICMP echo reply, id 302, seq 20, length 64
16:16:05.060215 IP 172.20.0.1 > 172.20.0.2: ICMP echo request, id 302, seq 21, length 64
16:16:05.060238 IP 172.20.0.2 > 172.20.0.1: ICMP echo reply, id 302, seq 21, length 64
16:16:06.084170 IP 172.20.0.1 > 172.20.0.2: ICMP echo request, id 302, seq 22, length 64
16:16:06.084187 IP 172.20.0.2 > 172.20.0.1: ICMP echo reply, id 302, seq 22, length 64

-> I do not see any iptables rules that may be forcing the packets to be NAT’ed to the Bridge’s IP address, but I may be missing something here. Can someone help me understand why this might be happening?

In summary:

  1. What is the exact flow of packets (A step by step description of how the packet traverses from the container, via the bridge and which iptables are checked for the packet)?

  2. Why is the source IP being NAT’ed when the packet reaches the destination? Is there any iptables rule that forces this to happen?

Thanks in advance!

The common apprach is to put both containers in the same custom network…

A running container can be added to an existing network like this:

docker network connect [OPTIONS] NETWORK CONTAINER

And disconnected like this:

docker network disconnect [OPTIONS] NETWORK CONTAINER

Docker-compose.yml based deployments allow a container to be in more than a single nework…

Hi meyay@,

Thanks for the response. Agreed that the common approach is to put containers that need to communicate in the same network, but I am trying different architectures. According to Docker docs, https://docs.docker.com/network/bridge/, it says the following:

“The Docker bridge driver automatically installs rules in the host machine so that containers on different bridge networks cannot communicate directly with each other.”

-> It is these rules that I am trying to play with. Any ideas as to what the exact flow of packets from source to destination is and why I am seeing the behavior as specified above?

Thanks again!

In te last 4 years i worked with Docker, I never had to… not even on deployements with containers beeing members of severall network :slight_smile:

Someone else has to pitch in.