We are seeing an issue in our system where the incoming tcp packets to docker container hit the DOCKER chain, but not DOCKER-USER. The receiving host never responds to the nping tcp request most likely due to the targeting container never receiving the packet. Here is the detail:
From host-2:
host-2 :~# iptables -t nat -L DOCKER -v -n | grep 3888
5049 261K DNAT tcp – !bridge_net * 0.0.0.0/0 10.20.100.110 tcp dpt:3888 to:169.254.8.10:3888
Now do nping from host-1:
host-1 :~# nping --tcp --dest-port 3888 --dest-ip 10.20.100.110 -e eth2
host-2:~# iptables -t nat -L DOCKER -v -n | grep 3888
5055 262K DNAT tcp – !bridge_net * 0.0.0.0/0 10.20.100.110 tcp dpt:3888 to:169.254.8.10:3888
As one can see, this DNAT rule below was hit when nping is issued from host-1:
-A DOCKER -d 10.20.100.110/32 ! -i bridge_net -p tcp -m tcp --dport 3888 -j DNAT --to-destination 169.254.8.10:3888
This means that the container should have received all 5 packets but never replied.
However, I do not see any hit on DOCKER-USER chain where I put some iptables rules on the DOCKER-USER chain and they get hit in the working case:
host-2:~# iptables -L DOCKER-USER -v -n | grep eth2
0 0 ACCEPT tcp – eth2 * 10.20.100.112 0.0.0.0/0
0 0 ACCEPT tcp – eth2 * 10.20.100.111] 0.0.0.0/0
0 0 ACCEPT all – eth2 * 0.0.0.0/0 0.0.0.0/0 state RELATED,ESTABLISHED
0 0 ACCEPT all – * eth2 0.0.0.0/0 0.0.0.0/0
0 0 LOGGING tcp – eth2 * 0.0.0.0/0 0.0.0.0/0
This evidence is telling me that no incoming packets received are going through the DOCKER-USER chain at all.
How could this happen?