I have a docker host with multiple IPs on one interface that can be reached from three ISP / Internet connections. Each gateway’s NAT configuration points to one of the host IPs and source based policy routing sorts out which gateway to use for return traffic. This works fine for native services such as SSH but not with a web server in a Docker container. The web server container always uses the default gateway regardless regardless of which host IP address (I.E. ISP connection) the traffic originates from. I have not been able to find any examples of how to configure / integrate a Docker container with source based routing. Any hints or pointers will be much appreciated.
I never dealt with this kind of setup before. But you might want to check Use macvlan networks | Docker Documentation and Use IPvlan networks | Docker Documentation if they can be part of a solution. Both require to specify a host interface (inclusing virtual interfaces), which will be used as breakout from the container to the particular host interface.
I can see how this would be an issue for egress traffic, where the request originates from the container to access a remote target, but not for ingress traffic, where a remote source receives a reponse to its request. But maybee I am mistaken about this, because I just never saw it different than that.
Can you tell more about your network settings? Do you have something like this?
10.1.1.2/24 --> ETHERNET 1 --> ROUTER 1 --> Server 1 on 192.168.1.2 --> Container 1 10.1.2.2/24 --> ETHERNET 2 --> ROUTER 1 --> Server 1 on 192.168.1.3 --> Container 2 10.1.3.2/24 --> ETHERNET 3 --> ROUTER 1 --> Server 1 on 192.168.1.4 --> Container 3
where you want the container to respond like this:
Container 1 --> 10.1.1.1 --> 10.1.1.2 Container 2 --> 10.1.2.1 --> 10.1.2.2 Container 3 --> 10.1.3.1 --> 10.1.3.2
Source based routing was new to me, so I read this: Source-based routing
Maybe the problem is that Docker containers have their own gateways and Docker routes the traffic to the default gateway before the policy could do anything (just a guess, I am not really a network guy)
You could try running the container on host network if you can controll what IP the service is listening on in the container.
You can also try what @meyay suggested if you can change the routing on the other side to point to the LAN IP address of the container.
Ther may be a better way, but if you find it, please, share it This is an interesting use case and I am curious what the solution will be.
That response can go thorugh the default gateway of the destination server which might not have access to the source, but the destination server which sends the response can’t tell that. For example the default gateway is a router that does not have internet access. Of course then the default route should be changed, but this was just an example, because I made that mistake before.
@cooperwa can you also tell more about your goal? I am not sure I understood everything. Maybe you don’t need that source based policy to achieve your goal and we can suggest an other solution.