How to open firewalld ports for Ingress networking on RHEL/CentOS 7?

Docker Engine Swarm
1

I have a 3 controller + 4 worker node swarm (Engine 1.12.1 in swarm mode on RHEL 7.2). I’m not having any luck figuring out how to configure firewalld to enable access to published services on the swarm.

As a test, I created an nginx service published on port 32000.

docker service create -p 32000:80 --name nginx nginx

On each individual node, I can curl http://127.0.0.1:32000/ and get the expected response. But, I can’t connect from any other machine. I just get curl: (7) Failed to connect to example.ucalgary.ca port 32000: Connection refused.

Firewalld is configured to accept connections on ports 30000 to 32767. If I turn firewalld off, then everything works. But, I can’t figure out how to get it to work with firewalld on. What am I misssing?

Here’s my firewalld service for docker-ingress.

<?xml version="1.0" encoding="utf-8"?>
<service>
  <short>Docker Ingress Load Balancing</short>
  <description>These ports enable Docker ingress load balancing in a Docker swarm. The default range for published ports is 30000 to 32767. See https://docs.docker.com/engine/swarm/key-concepts/ for details.</description>
  <port protocol="tcp" port="30000-32767"/>
</service>

And, the firewall config.

[root@example kchuang]# firewall-cmd --zone=public --list-all
public (default, active)
  interfaces: ens192
  sources: 
  services: docker-ingress docker-overlay
  ports: 
  masquerade: no
  forward-ports: 
  icmp-blocks: 
  rich rules: 
	rule family="ipv4" source address="10.0.0.0/8" service name="docker-ingress" accept
	rule family="ipv4" source address="136.159.0.0/16" service name="docker-ingress" accept
2

I’m seeing errors related to Docker and iptables in firewalld.service. I’m not sure what to do about this, or if it’s related.

[root@example kchuang]# systemctl status -l firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2016-10-03 12:19:28 MDT; 11min ago
 Main PID: 59262 (firewalld)
   Memory: 17.1M
   CGroup: /system.slice/firewalld.service
           └─59262 /usr/bin/python -Es /usr/sbin/firewalld --nofork --nopid

Oct 03 12:23:15 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:15 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -i docker0 -o docker_gwbridge -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Oct 03 12:23:15 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:15 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t filter -C DOCKER-ISOLATION -i docker_gwbridge -o docker0 -j DROP' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Oct 03 12:23:16 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:16 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t mangle -C OUTPUT -p udp --dport 4789 -m u32 --u32 0>>22&0x3C@12&0xFFFFFF00=65536 -j MARK --set-mark 13681891' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:46 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:46 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -L DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:46 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:46 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C DOCKER-INGRESS -j RETURN' failed: iptables: Bad rule (does a matching rule exist in that chain?).
Oct 03 12:23:46 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:46 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C OUTPUT -m addrtype --dst-type LOCAL -j DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:46 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:46 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C PREROUTING -m addrtype --dst-type LOCAL -j DOCKER-INGRESS' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:47 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:47 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -m addrtype --src-type LOCAL -o docker_gwbridge -j MASQUERADE' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:47 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:47 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -m ipvs --ipvs -d 10.255.0.0/16 -j SNAT --to-source 10.255.0.2' failed: iptables: No chain/target/match by that name.
Oct 03 12:23:47 itpocnode01.ucalgary.ca firewalld[59262]: 2016-10-03 12:23:47 ERROR: COMMAND_FAILED: '/sbin/iptables -w2 -t nat -C POSTROUTING -m ipvs --ipvs -d 10.255.0.0/16 -j SNAT --to-source 10.255.0.2' failed: iptables: No chain/target/match by that name.
3

Based on docker/docker#16816 and docker/docker#16137, it looks like some sort of issue with Docker and firewalld.

When I create a service, firewalld on the host that’s running the service shows:

2016-10-03 12:37:30 ERROR: COMMAND_FAILED: ‘/sbin/iptables -w2 -t nat -C POSTROUTING -m ipvs --ipvs -d 10.255.0.0/16 -j SNAT --to-source 10.255.0.8’ failed: iptables: No chain/target/match by that name.

So, how do I solve this? There isn’t an obvious answer in the two linked issues for swarm mode, as far as I can tell.

4

Enabling masquerade on firewalld made everything work. I still see the same errors in the logs about iptables, but my test swarm services are working with the ingress networking, now.

The solution in my case was:

firewall-cmd --zone=public --permanent --add-masquerade

2 Likes
5

I had exactly the same issue, your solution saved my day, THANKS!