For a while now I’ve been facing problems connecting to the outside world from within my images. I think I have pin-pointed the root cause of this. First let me describe the situation:
- Ubuntu server, running 14.04.2 LTS
- Two physical NICs: eth0 (internet) and p1p1 (local)
- The server is acting as a router for my internal network. The configuration of the iptables can be found below. Of course
/proc/sys/net/ipv4/ip_forwardis set to TRUE. - in
/etc/defaults/dockerI have set the following options:DOCKER_OPTS="--ip-masq=true --dns 8.8.8.8 --dns 8.8.4.4"
Reproducing the problem
Now when I start an image, f.e. this one: docker run -i -t ubuntu:14.04 /bin/bash and I’m trying to ping google.com, I receive a destination unknown. Some Googling taught me that it has to do with masqueradring not being enabled. So I checked the rules by running this: sudo iptables -t nat -L -n. And indeed, the output is as follows:
> ~/ -> sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
Basically this means that there is no NAT possible between a Docker container and “the internet”. Now when I restart Docker (sudo service docker restart), the missing rules are added:
> ~/ -> sudo iptables -t nat -L -n
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
DOCKER all -- 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
MASQUERADE all -- 172.17.0.0/16 0.0.0.0/0
MASQUERADE all -- 0.0.0.0/0 0.0.0.0/0
Chain DOCKER (2 references)
target prot opt source destination
My conclusion
My conclusion is that there is some kind of problem in the order at bootup. I suppose the docker daemon get’s started first (and adds the missing rules) and THEN the iptables are being restored. This is done through the /etc/rc.local file: iptables-restore < /etc/iptables.sav. I also find this post, which basically describes my problem, but in fewer words 
http://stackoverflow.com/questions/25917941/docker-how-to-re-create-dockers-additional-iptables-rules
IPTABLES configuration script
#!/bin/sh
# p1p1 = internal NIC
# eth0 = WAN NIC
lan_nic=p1p1
wan_nic=eth0
echo Configuration: [lan_nic = $lan_nic], [wan_nic = $wan_nic]
echo "Clear old firewall rules..."
iptables --flush
iptables --flush FORWARD
iptables --flush INPUT
iptables --flush OUTPUT
iptables --table nat --flush
iptables --table nat --delete-chain
iptables --table mangle --flush
iptables --table mangle --delete-chain
iptables --delete-chain
echo "Drop all INPUT and FORWARD..."
iptables -P INPUT DROP
iptables -P FORWARD DROP
echo "Drop all IPv6 traffic..."
ip6tables -P INPUT DROP
ip6tables -P OUTPUT DROP
ip6tables -P FORWARD DROP
echo "Accept everything on lo and $lan_nic (local network)..."
iptables -A INPUT -i lo -p all -j ACCEPT
iptables -A OUTPUT -o lo -p all -j ACCEPT
iptables -A INPUT -i $lan_nic -p all -j ACCEPT -s 192.168.1.0/24
iptables -A OUTPUT -o $lan_nic -p all -j ACCEPT -d 192.168.1.0/24
echo "IP Forwarding and Routing for gateway use..."
iptables -A FORWARD -o $wan_nic -i $lan_nic -s 192.168.1.0/24 -m conntrack --ctstate NEW -j ACCEPT
iptables -A FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables -t nat -A POSTROUTING -o $wan_nic -j MASQUERADE
echo "Maintain established connections..."
iptables -A INPUT --in-interface $wan_nic --match conntrack --ctstate ESTABLISHED,RELATED --jump ACCEPT
echo "Allow port 80 (http) connections to the firewall..."
iptables -A INPUT -i $wan_nic -p tcp --dport 80 -m state --state NEW -j ACCEPT