Hello everybody,
The context
I got two docker-containers:
- Container #1 is hosting an open source ticketing system
- Container #2 is hosting a mailserver
Container #1 is connecting to Container #2 by using a globally resolvable hostname that points to the docker-host.
The mailserver is up and running and I can connect to it using my local PC and a default mailclient.
d53cb727de95 tvial/docker-mailserver "supervisord -c /etc…" 4 days ago Up 3 minutes 0.0.0.0:25->25/tcp, :::25->25/tcp, 110/tcp, 0.0.0.0:143->143/tcp, :::143->143/tcp, 0.0.0.0:587->587/tcp, :::587->587/tcp, 465/tcp, 995/tcp, 0.0.0.0:993->993/tcp, :::993->993/tcp, 4190/tcp mail
Using nmap on my local computer I can see that the port is open wide:
PORT STATE SERVICE
993/tcp open imaps
iptables got these rules:
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N f2b-sshd
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-A INPUT -p tcp -m multiport --dports 22 -j f2b-sshd
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -j DOCKER
-A FORWARD -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -i br-fe03c792b7c6 -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -j DOCKER
-A FORWARD -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -j DOCKER
-A FORWARD -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -i br-12fecdadfb26 -o br-12fecdadfb26 -j ACCEPT
-A f2b-sshd -s 95.236.224.201/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-fe03c792b7c6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-8d0b3e4c5cd8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-12fecdadfb26 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
The problem
As I am using puppet to manage this server, I created a module that manages iptables for me. As docker is terrible to use this way, I had to write some rules to ignore those docker-related rules.
This is the result of my puppetized iptables:
-P INPUT DROP
-P FORWARD DROP
-P OUTPUT DROP
-N f2b-sshd
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N DROP_INPUT
-N DROP_OUTPUT
-N DROP_FORWARD
-A INPUT -i lo -m comment --comment "000 allow incoming on loopback" -j ACCEPT
-A INPUT -i lo -m comment --comment "000 allow incoming on loopback IPv6" -j ACCEPT
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN FIN,SYN -m comment --comment "005 drop bogus fin,syn" -j DROP_INPUT
-A INPUT -p tcp -m tcp --tcp-flags SYN,RST SYN,RST -m comment --comment "006 drop bogus syn,rst" -j DROP_INPUT
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "007 allow incoming established, related" -j ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -m comment --comment "007 allow incoming established, related IPv6" -j ACCEPT
-A INPUT -p icmp -m comment --comment "008 allow incoming icmp echo-requests" -j ACCEPT
-A INPUT -p ipv6-icmp -m comment --comment "008 allow incoming icmp echo-requests IPv6" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "020 allow incoming ssh" -j ACCEPT
-A INPUT -p tcp -m multiport --dports 22 -m comment --comment "020 allow incoming ssh IPv6" -j ACCEPT
-A INPUT -m comment --comment "999 drop all incoming" -j DROP_INPUT
-A INPUT -m comment --comment "999 drop all incoming IPv6" -j DROP_INPUT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-fe03c792b7c6 -j DOCKER
-A FORWARD -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -i br-fe03c792b7c6 -o br-fe03c792b7c6 -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-8d0b3e4c5cd8 -j DOCKER
-A FORWARD -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-12fecdadfb26 -j DOCKER
-A FORWARD -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -i br-12fecdadfb26 -o br-12fecdadfb26 -j ACCEPT
-A FORWARD -m comment --comment "999 drop all forwarding" -j DROP_FORWARD
-A FORWARD -m comment --comment "999 drop all forwarding IPv6" -j DROP_FORWARD
-A OUTPUT -o lo -m comment --comment "000 allow outgoing on loopback" -j ACCEPT
-A OUTPUT -o lo -m comment --comment "000 allow outgoing on loopback IPv6" -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -m comment --comment "005 allow outgoing established, related" -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -m comment --comment "005 allow outgoing established, related IPv6" -j ACCEPT
-A OUTPUT -m state --state NEW -m comment --comment "006 allow new outgoing" -j ACCEPT
-A OUTPUT -m state --state NEW -m comment --comment "006 allow new outgoing IPv6" -j ACCEPT
-A OUTPUT -m comment --comment "999 drop all outgoing" -j DROP_OUTPUT
-A OUTPUT -m comment --comment "999 drop all outgoing IPv6" -j DROP_OUTPUT
-A f2b-sshd -s 95.236.224.201/32 -j REJECT --reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 993 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 587 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 143 -j ACCEPT
-A DOCKER -d 172.22.0.2/32 ! -i br-fe03c792b7c6 -o br-fe03c792b7c6 -p tcp -m tcp --dport 25 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 8080 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 443 -j ACCEPT
-A DOCKER -d 192.168.64.4/32 ! -i br-8d0b3e4c5cd8 -o br-8d0b3e4c5cd8 -p tcp -m tcp --dport 80 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-fe03c792b7c6 ! -o br-fe03c792b7c6 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-8d0b3e4c5cd8 ! -o br-8d0b3e4c5cd8 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i br-12fecdadfb26 ! -o br-12fecdadfb26 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-fe03c792b7c6 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-8d0b3e4c5cd8 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o br-12fecdadfb26 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A DROP_INPUT -m limit --limit 10/min -m comment --comment "000 log dropped INPUT IPv4" -j LOG --log-prefix "iptables DROP_INPUT: " --log-level 7
-A DROP_INPUT -m comment --comment "001 drop INPUT IPv4" -j DROP
-A DROP_OUTPUT -m limit --limit 10/min -m comment --comment "000 log dropped OUTPUT IPv4" -j LOG --log-prefix "iptables DROP_OUTPUT: " --log-level 7
-A DROP_OUTPUT -m comment --comment "001 drop OUTPUT IPv4" -j DROP
-A DROP_FORWARD -m limit --limit 10/min -m comment --comment "000 log dropped FORWARD IPv4" -j LOG --log-prefix "iptables DROP_FORWARD: " --log-level 7
-A DROP_FORWARD -m comment --comment "001 drop FORWARD IPv4" -j DROP
This is working, too - for my local computer. But having these rules enabled, Container #1 can no longer connect to Container #1 (host unreachable) and I do not see the reason.