Accessing containers in a multi-homed docker host

Hi

I am trying to setup containers in a multi-homed host and only publishing the contianers on specific interfaces. This has proven to be a challenge for me, I guess it’s routing issues.

I have four nics on the host, ens160, ens192, ens224 and docker0. ens160 has the default gw.

I’ve setup policy based routing to make it possible to access services on ens192 from hosts on another subnet which is also supposed to be able to reach services published on ens160.

The ips on my host:
ens160 10.101.3.250/24
ens192 10.101.10.250/24
ens224 10.101.100.250/24
docker0 172.17.0.1/16

main routing table is pretty standard with default gw on 10.101.3.254

routing table ens192 has the following rule and routes:
ip rule add from 10.101.10.250 table ens192
ip route add 10.101.1.0/24 via 10.101.10.254 table ens192

The routing works just fine for pinging the host on ens192 and reaching ssh. So far all the working is great.

But reaching containers does not work at all. I’ve set up portainer to listen to all interfaces for testing, but the goal is to only publish protainer on ens192.

Trying to reach portainer via ens160 works great, and packets are forwarded properly to the docker0 interface.

But nothing at all happens on the docker0 interface when I try to reach portainer via ens192.

I don’t see where the packets disapear. I can see the syn packets getting in on ens192, but there it stops, no syn packets on docker0.

As said trying to ping or ssh to ens192 works fine and I can see that the policy based routing for ens192 kicks in perfectly. So my guess is that docker-proxy gets confused but I can’t figure out why, even less figure out how to solve.

Anyone?

/Peter

If you are running on Linux and have not looked at your iptables (firewall) configuration you might want to. While my set up differs from yours, I burned absurd amounts of time thinking routing and other things were the issue, and it ended up just being a couple iptables rules breaking everything.

Good luck.


Scot

Thanks Scot! Is it the iptables rules that are auto-generated by Docker that you’re referring to? I haven’t created any iptables rules of my own on this host.

KR,
Peter

My set up was perhaps more messy using VMs and such, so I am not sure who/what created the iptables rules because I’d not even looked at them before setting up all the containers and starting to try to figure out what was going wrong.

Your experience of ping/ssh working but other traffic not working is similar to what I’d experienced which made me want to warn you about iptables.

In case it’s of any help…my issues were:

  1. the default gateway for the route I’d added for my container’s interface wasn’t getting set up correctly, it needed a separate entry where a docker interface was used specifically for the gateway in order to get the request through to the inner VM (again not sure this would in any way be applicable to your situation)

  2. iptables settings had a rule or two that was blocking things.

By using wireshark externally, and tcpdump internally (on the server), I could finally see there were issues with ARPs not being responded to, which lead me to what was going on with the first issue.

After solving the first issue and being able to see that requests were making it to the interfaces and yet still failing, the only thing left seemed to be firewall rules. Copied my iptables setting, and then removed all iptables rules just to confirm whether or not it was the issue, and it was. Then I put the rules back one by one until it broke again, and that helped me understand what the rules were doing and what alterations needed to be made.

Just had wanted to warn you about the potential for iptables to trip you up. Sorry I can’t be or more help, this was probably close to 2 years ago, and there are so many different ways to use docker that it can be a lot harder than one would hope.

thanks for your helpful answers. i was having the same question as posterberghexbit. I really need some help with my main course project. i will try your methods and may come with updates.

Thank you very much!