Hello Team,
I’m having difficulties in finding out RCA for why there’s not internet access in my docker containers available in kubernetes cluster. I have changed iptable rules multiple times, did R&Ds with many aspects - but still no luck -
Please let me know what to change in IPtable - what to remove most probably to make things good.
My iptable -
============================================================
Generated by iptables-save v1.6.0 on Wed Dec 16 06:28:01 2020
Completed on Wed Dec 16 06:28:01 2020
Generated by iptables-save v1.6.0 on Wed Dec 16 06:28:01 2020
*mangle
:PREROUTING ACCEPT [69214835:19699083607]
:INPUT ACCEPT [69205268:19698333736]
:FORWARD ACCEPT [3:185]
:OUTPUT ACCEPT [68398212:14079006675]
:POSTROUTING ACCEPT [68398215:14079006860]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
COMMIT
Completed on Wed Dec 16 06:28:01 2020
Generated by iptables-save v1.6.0 on Wed Dec 16 06:28:01 2020
*filter
:INPUT ACCEPT [659:67196]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [61515:15720063]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
:KUBE-EXTERNAL-SERVICES - [0:0]
:KUBE-FIREWALL - [0:0]
:KUBE-FORWARD - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SERVICES - [0:0]
-A INPUT -m conntrack --ctstate NEW -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A INPUT -m conntrack --ctstate NEW -m comment --comment “kubernetes externally-visible service portals” -j KUBE-EXTERNAL-SERVICES
-A INPUT -s 127.0.0.1/32 -m comment --comment “Allowing all local incoming connections” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 80 -m comment --comment “Enabling http traffic for all sources” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 443 -m comment --comment “Enabling https traffic for all sources” -j ACCEPT
-A INPUT -s 61.12.67.16/29 -p tcp -m tcp --dport 1122 -m comment --comment “Enabling ssh traffic for cavisson sources” -j ACCEPT
-A INPUT -s 182.71.119.208/28 -p tcp -m tcp --dport 1122 -m comment --comment “Enabling ssh traffic for cavisson sources” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 7891 -m comment --comment “Enabling CMON traffic” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 7892 -m comment --comment “Enabling ndc traffic” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 8000:8010 -m comment --comment “Enabling http for multiple controllers” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 623 -m comment --comment “Enabling IPMI” -j ACCEPT
-A INPUT -s 61.12.67.16/29 -p tcp -m tcp --dport 5900:5950 -m comment --comment “redundant till port range 1024-65000 is active” -j ACCEPT
-A INPUT -s 61.12.67.16/29 -p tcp -m tcp --dport 6000:6050 -m comment --comment “redundant till port range 1024-65000 is active” -j ACCEPT
-A INPUT -s 182.71.119.208/28 -p tcp -m tcp --dport 5900:5950 -m comment --comment “redundant till port range 1024-65000 is active” -j ACCEPT
-A INPUT -s 182.71.119.208/28 -p tcp -m tcp --dport 6000:6050 -m comment --comment “redundant till port range 1024-65000 is active” -j ACCEPT
-A INPUT -p tcp -m tcp --dport 1025:65535 -m comment --comment “Allowing NS Traffic” -j ACCEPT
-A INPUT -p tcp -m tcp --sport 53 -m comment --comment “Allowing DNS” -j ACCEPT
-A INPUT -p icmp -m icmp --icmp-type 8 -m state --state NEW,RELATED,ESTABLISHED -m comment --comment “Allowing ICMP Echo input” -j ACCEPT
-A INPUT -p tcp -m comment --comment “Rejecting all connections” -j REJECT --reject-with icmp-port-unreachable
-A FORWARD -m comment --comment “kubernetes forwarding rules” -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A OUTPUT -s 127.0.0.1/32 -m comment --comment “Allowing all local outgoing connections” -j ACCEPT
-A OUTPUT -p icmp -m icmp --icmp-type 0 -m state --state RELATED,ESTABLISHED -m comment --comment “Allowing ICMP Echo output” -j ACCEPT
-A KUBE-EXTERNAL-SERVICES -p tcp -m comment --comment “kubernetes-dashboard/kubernetes-dashboard has no endpoints” -m addrtype --dst-type LOCAL -m tcp --dport 30000 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment “kubernetes forwarding rules” -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment “kubernetes forwarding conntrack pod source rule” -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-FORWARD -m comment --comment “kubernetes forwarding conntrack pod destination rule” -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment “kube-system/kube-dns:metrics has no endpoints” -m tcp --dport 9153 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p tcp -m comment --comment “kube-system/kube-dns:dns-tcp has no endpoints” -m tcp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.108.36.36/32 -p tcp -m comment --comment “kubernetes-dashboard/dashboard-metrics-scraper has no endpoints” -m tcp --dport 8000 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.231.116/32 -p tcp -m comment --comment “kubernetes-dashboard/kubernetes-dashboard has no endpoints” -m tcp --dport 443 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.96.0.10/32 -p udp -m comment --comment “kube-system/kube-dns:dns has no endpoints” -m udp --dport 53 -j REJECT --reject-with icmp-port-unreachable
-A KUBE-SERVICES -d 10.101.112.23/32 -p tcp -m comment --comment “kube-system/metrics-server:https has no endpoints” -m tcp --dport 443 -j REJECT --reject-with icmp-port-unreachable
COMMIT
Completed on Wed Dec 16 06:28:01 2020
Generated by iptables-save v1.6.0 on Wed Dec 16 06:28:01 2020
*nat
:PREROUTING ACCEPT [0:0]
:INPUT ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
:POSTROUTING ACCEPT [0:0]
:CNI-DN-01ce522568bc825e47f93 - [0:0]
:CNI-DN-2117955e874db3e74688d - [0:0]
:CNI-DN-cd39fa1b7f42f77e4fd99 - [0:0]
:CNI-DN-d3d87d7305e82eb421fa7 - [0:0]
:CNI-HOSTPORT-DNAT - [0:0]
:CNI-HOSTPORT-MASQ - [0:0]
:CNI-HOSTPORT-SETMARK - [0:0]
:DOCKER - [0:0]
:KUBE-KUBELET-CANARY - [0:0]
:KUBE-MARK-DROP - [0:0]
:KUBE-MARK-MASQ - [0:0]
:KUBE-NODEPORTS - [0:0]
:KUBE-POSTROUTING - [0:0]
:KUBE-PROXY-CANARY - [0:0]
:KUBE-SEP-JIQV5AQRXNWBERNO - [0:0]
:KUBE-SERVICES - [0:0]
:KUBE-SVC-NPX46M4PTMTKRN6Y - [0:0]
-A PREROUTING -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A PREROUTING -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT -m comment --comment “kubernetes service portals” -j KUBE-SERVICES
-A OUTPUT -m addrtype --dst-type LOCAL -j CNI-HOSTPORT-DNAT
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -m comment --comment “CNI portfwd requiring masquerade” -j CNI-HOSTPORT-MASQ
-A POSTROUTING -m comment --comment “kubernetes postrouting rules” -j KUBE-POSTROUTING
-A POSTROUTING -s 192.168.0.0/20 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A POSTROUTING -s 10.244.0.0/16 -d 10.244.0.0/16 -j RETURN
-A POSTROUTING -s 10.244.0.0/16 ! -d 224.0.0.0/4 -j MASQUERADE
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/24 -j RETURN
-A POSTROUTING ! -s 10.244.0.0/16 -d 10.244.0.0/16 -j MASQUERADE
-A CNI-DN-01ce522568bc825e47f93 -s 10.244.0.0/24 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-01ce522568bc825e47f93 -s 127.0.0.1/32 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-01ce522568bc825e47f93 -p tcp -m tcp --dport 7891 -j DNAT --to-destination 10.244.0.13:7891
-A CNI-DN-2117955e874db3e74688d -s 10.244.0.0/24 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2117955e874db3e74688d -s 127.0.0.1/32 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-2117955e874db3e74688d -p tcp -m tcp --dport 7891 -j DNAT --to-destination 10.244.0.17:7891
-A CNI-DN-cd39fa1b7f42f77e4fd99 -s 10.244.0.0/24 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-cd39fa1b7f42f77e4fd99 -s 127.0.0.1/32 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-cd39fa1b7f42f77e4fd99 -p tcp -m tcp --dport 7891 -j DNAT --to-destination 10.244.0.7:7891
-A CNI-DN-d3d87d7305e82eb421fa7 -s 10.244.0.0/24 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-d3d87d7305e82eb421fa7 -s 127.0.0.1/32 -p tcp -m tcp --dport 7891 -j CNI-HOSTPORT-SETMARK
-A CNI-DN-d3d87d7305e82eb421fa7 -p tcp -m tcp --dport 7891 -j DNAT --to-destination 10.244.0.5:7891
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment “dnat name: "cbr0" id: "46209358a4c52594b87c181aa3968d15e5269a2f7006d78288a37163ecaa8b26"” -m multiport --dports 7891 -j CNI-DN-cd39fa1b7f42f77e4fd99
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment “dnat name: "cbr0" id: "fe9172c0e3ecbed13b8e7a7649e12290cb1858d4de11beea288836ecaf28d2dc"” -m multiport --dports 7891 -j CNI-DN-2117955e874db3e74688d
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment “dnat name: "cbr0" id: "4d33488b9c5b7309645166792c8cd504b8873e714a6e1101de7f87f922e93a1f"” -m multiport --dports 7891 -j CNI-DN-01ce522568bc825e47f93
-A CNI-HOSTPORT-DNAT -p tcp -m comment --comment “dnat name: "cbr0" id: "0def6f14850c73f67fc6914735fe52ae06c339789454a6320a5ee1d59d83f747"” -m multiport --dports 7891 -j CNI-DN-d3d87d7305e82eb421fa7
-A CNI-HOSTPORT-MASQ -m mark --mark 0x2000/0x2000 -j MASQUERADE
-A CNI-HOSTPORT-SETMARK -m comment --comment “CNI portfwd masquerade mark” -j MARK --set-xmark 0x2000/0x2000
-A DOCKER -i docker0 -j RETURN
-A KUBE-MARK-DROP -j MARK --set-xmark 0x8000/0x8000
-A KUBE-MARK-MASQ -j MARK --set-xmark 0x4000/0x4000
-A KUBE-POSTROUTING -m mark ! --mark 0x4000/0x4000 -j RETURN
-A KUBE-POSTROUTING -j MARK --set-xmark 0x4000/0x0
-A KUBE-POSTROUTING -m comment --comment “kubernetes service traffic requiring SNAT” -j MASQUERADE
-A KUBE-SEP-JIQV5AQRXNWBERNO -s 64.188.18.2/32 -m comment --comment “default/kubernetes:https” -j KUBE-MARK-MASQ
-A KUBE-SEP-JIQV5AQRXNWBERNO -p tcp -m comment --comment “default/kubernetes:https” -m tcp -j DNAT --to-destination 64.188.18.2:6443
-A KUBE-SERVICES ! -s 10.244.0.0/16 -d 10.96.0.1/32 -p tcp -m comment --comment “default/kubernetes:https cluster IP” -m tcp --dport 443 -j KUBE-MARK-MASQ
-A KUBE-SERVICES -d 10.96.0.1/32 -p tcp -m comment --comment “default/kubernetes:https cluster IP” -m tcp --dport 443 -j KUBE-SVC-NPX46M4PTMTKRN6Y
-A KUBE-SERVICES -m comment --comment “kubernetes service nodeports; NOTE: this must be the last rule in this chain” -m addrtype --dst-type LOCAL -j KUBE-NODEPORTS
-A KUBE-SVC-NPX46M4PTMTKRN6Y -m comment --comment “default/kubernetes:https” -j KUBE-SEP-JIQV5AQRXNWBERNO
COMMIT
Completed on Wed Dec 16 06:28:01 2020
=================================================================