Docker-compose split routing with physical interfaces

General
1

Hello.

My homelab server has two physical interfaces. I’ve named them vpn and wan respectively within the host OS. My router is setup as such that packets with a source address of the IP assigned to wan go out through my ISP gateway, whereas packets with a source destination of the IP assigned to vpn go out through my openvpn gateway.

I have multiple docker containers on my homelab server and I wished to send traffic from some (namely game servers) through my wan connection and other containers out through the vpn connection.

HOWEVER, I’ve continuously had troubles getting this to work. It seems like it would be an easy thing to do. I’ve attached an illustration of my setup and the test I put together to show my bad results. The host results are also within the image showing that routing does work properly outside of the containers.

Split.Interface.Routing.w.Docker
Split.Interface.Routing.w.Docker1642×931 73.9 KB

Ultimately the questions are
is my docker-compose test file right?
does com.docker.network.host_binding_ipv4 even work?
is docker somehow not correctly NAT’ing the given networks with the actual correct IP?

Does anyone have any ideas?

2

Hi, please let me know when you get this to work property. I have tried this with docker desktop and wls2 containers without success.

3

Let me see if I remember all of the details on how I am now doing this. It’s not perfect, but it is pretty close.

From a high-level concept view:

  • HOST OS
  • I use iptables to manually add two routes to for iptable vpn and wan
  • I use an iptables PREROUTING rule on all packets destined for a Network vpn to MARK them with a 1
  • I use iptables rule to use the iptable vpn with packets marked with 1
  • DOCKER
  • docker-compose: I define two networks respectively wan_traffic and vpn_traffic with hard defined networks and I used com.docker.network.bridge.host_binding_ipv4 to bind the networks respectively to the IP addresses matching my wan nic and my vpn nic.

docker-compose yaml:

  wan_traffic:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: wan_traffic
      com.docker.network.bridge.enable_ip_masquerade: "true"
      com.docker.network.bridge.host_binding_ipv4: "192.168.5.10"
      com.docker.network.enable_ipv6: "false"
    ipam:
      config:
        - subnet: 172.1.1.0/27

  vpn_traffic:
    driver: bridge
    driver_opts:
      com.docker.network.bridge.name: vpn_traffic
      com.docker.network.bridge.enable_ip_masquerade: "true"
      com.docker.network.bridge.host_binding_ipv4: "192.168.10.10"
      com.docker.network.enable_ipv6: "false"
    ipam:
      config:
        - subnet: 172.1.1.32/27

system startup script

ip route add 192.168.10.0/24 dev vpn table vpn
ip route add 0/0 via 192.168.10.1 dev vpn table vpn
iptables -t mangle -I PREROUTING -s 172.1.1.32/27 -j MARK --set-mark 1
ip rule add from all fwmark 1 table vpn

Hopefully this is enough to help :slight_smile:

1 Like