Docker Container Not Accessible from VLAN on Same Interface as Non VLAN

General
1

I have a sever with multiple network cards and I’m having trouble access exposed container ports from the vlan interface. Here is what the network looks like. I can successfully connect to the container on 127.0.0.1, 10.0.2.17 and 10.0.4.17 but not 10.0.3.17. I’m assuming it has something to do with multiple networks on the same card but I haven’t found a workaround. I ran into this issue with Kubernetes but tracked it down to the Docker level.

eth0,eth1
  bond0 - 802.3ad
    vlan14
      ip 10.0.4.17
eth2,eth3
  bond1 - active/backup
    ip 10.0.2.17
    vlan13
      ip 10.0.3.17

And here I am doing a test.

ubuntu@cloudmgr1:~ sudo docker run -d --rm --name web-test -p 8888:8000 crccheck/hello-world 4e59fb70b77084496f16b118c59815cdc38b327413268e30466236106035a198 ubuntu@cloudmgr1:~ netstat -tln | grep 8888
tcp6 0 0 :::8888 :::* LISTEN
ubuntu@cloudmgr1:~$ curl 10.0.2.17:8888

Hello World

                                   ##         .
                             ## ## ##        ==
                          ## ## ## ## ##    ===
                       /""""""""""""""""\___/ ===
                  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                       \______ o          _,/
                        \      \       _,'
                         `'--.._\..--''
ubuntu@cloudmgr1:~$ curl 10.0.3.17:8888 curl: (7) Failed to connect to 10.0.3.17 port 8888: No route to host ubuntu@cloudmgr1:~$ curl 10.0.4.17:8888 Hello World 
                                   ##         .
                             ## ## ##        ==
                          ## ## ## ## ##    ===
                       /""""""""""""""""\___/ ===
                  ~~~ {~~ ~~~~ ~~~ ~~~~ ~~ ~ /  ===- ~~~
                       \______ o          _,/
                        \      \       _,'
                         `'--.._\..--''
2

The issue seems to be caused by the ip routing policy that allows traffic on the 10.0.3.0 network to go out the 10.0.3.1 gateway. Without that rule a local apache http server won’t respond on the 10.0.3.17. I’m assuming I need another routing policy but I’m not sure what preventing Docker from seeing the traffic.

ip rule
0:	from all lookup local
0:	from 10.0.3.0/24 to 10.0.3.0/24 lookup main
100:	from 10.0.3.0/24 lookup 1
32766:	from all lookup main
32767:	from all lookup default

ip route show table main
default via 10.0.2.1 dev bond1 proto static
default via 10.0.3.1 dev bond1.13 proto static
10.0.2.0/24 dev bond1 proto kernel scope link src 10.0.2.11
10.0.3.0/24 dev bond1.13 proto kernel scope link src 10.0.3.11
10.0.4.0/24 dev bond0.14 proto kernel scope link src 10.0.4.11
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1

ip route show table 1
default via 10.0.3.1 dev bond1.13 proto static
3

Just wanted to post an update. I found this post which described how to do it. You have to add firewall marks so that IP Tables can figure out which interface the connection came in on. This assumes vlan 13 is an interface called vlan13. We tag all the traffic coming in with 0x13 and then put everything back when it leaves. You can follow the first 4 commands for each additional network you want to be able to respond on.

ip route add 10.0.3.0/24 dev vlan13 table 1013
ip route add 0/0 via 10.0.3.1 dev vlan13 table 1013
ip rule add fwmark 0x13 table 1013
iptables -t mangle -A PREROUTING -i vlan13 -m conntrack --ctstate NEW --ctdir ORIGINAL -j CONNMARK --set-mark 0x13

iptables -t mangle -A PREROUTING -m conntrack ! --ctstate NEW --ctdir REPLY -m connmark ! --mark 0x0 -j CONNMARK --restore-mark
iptables -t mangle -A OUTPUT -m conntrack ! --ctstate NEW --ctdir REPLY -m connmark ! --mark 0x0 -j CONNMARK --restore-mark