POSTROUTING SNAT not applied on all outgoing traffic

General
1

Hello everyone,

Issue type: I have incoming traffic on my router/firewalls LAN interface using the source IP of one of my docker containers internal IP: 172.17.0.3.

OS: Ubuntu 20.04.2 LTS

App version: Docker version 20.10.2, build 20.10.2-0ubuntu1~20.04.2

I find the issue quite peculiar. Everything seems to be fine. I’m running linuxserver.io:s qbittorrent container and it seems to run fine. The incoming port is exposed correctly and I can access everything. If i log into the container I can curl internet adresses just fine. Since I use another IP-range for my LAN all packets with this source IP is dropped by the firewall, i.e. I shouldnt have internet access at all if all packets hit the firewall with the incorrect source IP.

I checked the iptables nat tables postrouting chain, looks legit. The strange thing is most of the connections (95%) with the internal IP 172.17.0.3 have 63793 as source port. This is a port I have exposed on the host to the container, 5% use another port. It seems as if most traffic is getting SNAT:ed into the host IP since I can curl from within the container, connect to the exposed services and everything just seems to work fine.

Any ideas on how to further debug this issue?

iptables -t nat -S output:
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 63793 -j MASQUERADE
-A POSTROUTING -s 172.17.0.3/32 -d 172.17.0.3/32 -p tcp -m tcp --dport 8080 -j MASQUERADE
-A POSTROUTING -s 172.17.0.5/32 -d 172.17.0.5/32 -p tcp -m tcp --dport 8096 -j MASQUERADE
-A POSTROUTING -s 172.17.0.4/32 -d 172.17.0.4/32 -p tcp -m tcp --dport 4040 -j MASQUERADE
-A POSTROUTING -s 172.17.0.2/32 -d 172.17.0.2/32 -p tcp -m tcp --dport 445 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 63793 -j DNAT --to-destination 172.17.0.3:63793
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8080 -j DNAT --to-destination 172.17.0.3:8080
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 8096 -j DNAT --to-destination 172.17.0.5:8096
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 4040 -j DNAT --to-destination 172.17.0.4:4040
-A DOCKER ! -i docker0 -p tcp -m tcp --dport 445 -j DNAT --to-destination 172.17.0.2:445

3

Thank you for your answer.

That’s what makes it weird. The rule:

-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE

should as I understand make it impossible for packets to escape the host with the IP 172.17.0.3 still set. But when some packets hit my router that’s their source IP. Not all packets from the container experience this which is even weirder.

Maybe I need to tcpdump or conntrack this.

4

The story so far.

I added a “01-firewall_trace.conf” file in /etc/rsyslog.d/ containing

:rawmsg,contains,“TRACE” /var/log/firewall_trace.log
& stop

I restarted the rsyslog daemon

sudo systemctl restart rsyslog

Added a TRACE rule in iptables

sudo iptables -t raw -A OUTPUT -s 172.17.0.0/24 -j TRACE

and nothing shows up in the log file. If I add a catch all rule in OUTPUT the log is filled with trace messages but only with SRC=HOST IP.

Could this be because docker is bridging the nics? It seems as if packets from a bridged nic isn’t passing through the raw tables OUTPUT chain.

2412×790 310 KB

5

I can’t make heads or tail out of this. If anyone is experienced in docker networking I’d really appreciate a hand.

the docker0 bridge is not attached to the physical nic eth0, only the docker containers vnics are attached. The NAT POSTROUTING chain contains masquerading rules that process >99.9% of all traffic from the docker containers. Somehow, one container leaks packets with it’s internal IP 172.17.0.3 to my LAN.

tcpdump shows the packets leaving the eth0 interface. I tried adding a SNAT rule on the NAT POSTROUTING chain targeting source ip 172.17.0.3. The packets still leak. I’m pretty sure the packets do not traverse the iptables NAT POSTROUTING chain.

According to what I’ve understood of bridged nics this could be the case if the physical nic was attached to the bridge. Then a frame could be bridged without even being lifted to the iptables chains. But since my physical nic is NOT attached to the docker0 bridge the only way a fram/packet could leave the host is by being routed which would make it have to traverse the iptables NAT POSTROUTING chain. This seems also to be the case with >99.9% of all packets in docker.

Is there anyway to trace the way a packets is bridged and routed to figure out why these few packets are emitted from the host?