Issue with Multiple ENI on Aws

General
1

Two interfaces (eth0 and eth1) with two IP’s in different networks (the example IP’s we have used, 192.168.1.200 and 192.168.2.197) in a machine with any version of Ubuntu (16.04 or 18.04). Two containers, one (container1) with -p 192.168.1.200:8080:80 and another (container2) with -p 192.168.2.197:8080:80. The expected behavior is that it could successfully load http://publi_ip_container1 and http://publi_ip_container2.

Actual behavior

Once the traffic enters the machine, it is sent to the container through the docker0 bridge, at which point, when performing NAT for the docker network, it loses its initial origin. When the response is generated inside the container, it is sent out through the docker0 interface, but since it has lost its origin in the incoming NAT, the traffic will always be sent through the eth0. That is, the traffic coming from the container (response traffic) is always sent via eth0, despite this being received by eth1 (asymmetric routing).

Additional Information

Steps to reproduce the behavior

  1. We start from an AWS instance with docker-ce installed and with an elastic ip connected to the main private ip (eip1)
  2. We connect a second ENI to that instance with an elastic ip (eip2)
  3. We run a container with a listening port, e.g. port 80 (docker run -tid -p ip1:8080:80 image)
  4. From a browser we can load http://eip1:8080
  5. Now we start a second container (docker run -tid -p ip2:8080:80 image)
  6. In a browser load http://eip2:8080 and return a timeout

Some idea?

2

I doubt what you describe is limited to AWS and ENIs, it should be reproducible on every host that has more than a single network interface.

The docker0 interface should have an ip from a non routed network and shouldn’t be able to send traffic outside the host.

Since docker bridge networks are masqueraded/natted, I expected response traffic to originate on the host level from the ip of the interface the traffic came from, or at least has a route to that subnet. If both interfaces have a route to the target network and ip, than it will use the prefered interface.

It’s worrying that you reference Ubuntu versions that reached end of life years ago. Even Ubuntu 20.04 will reach end of life at the end of May (as in tomorrow)!

.