Overlay Network with Encryption and Changed Data Path Port

Docker Engine Swarm
1

Hi everyone,

I’m trying to set up a docker swarm and need a little help as I am somewhat lost. I plan on setting up three virtual machines (VMWare) in order to build a MQTT infrastructre. I am running into troubles with the encryption of the data plane (which is a requirement to have). It seems like as soon as I change the data path port in the swarm init (e.g. docker swarm init --data-path-port 4000) the encrypted communication (creating an overlay network with the option --opt encrypted=true) between docker hosts doesn’t work anymore. It works without encryption. It also works encrypted and unencrypted on my test system as long as I don’t change the data path port during initialization.

The issue is, I have to change it on my future production system as VMWare is sitting on the default data path port and blocks any communication. The system there works unencrypted when I change the port just like on my test system.

Tl;dr: Encrypted overlay networks stop working once i change the data path port. I need to change it though…

I have a minimal setup for my scenario:

version: "3.7"
services:
  vmq0:
    image: vernemq/vernemq:1.12.3
    hostname: 'vmq0'
    environment:
      - DOCKER_VERNEMQ_SWARM=1
      - DOCKER_VERNEMQ_ACCEPT_EULA=yes
      - DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on
      - DOCKER_NET_INTERFACE=eth0
      - DOCKER_VERNEMQ_LISTENER__TCP__DEFAULT=0.0.0.0:1883
      - DOCKER_VERNEMQ_DISTRIBUTED_COOKIE=vmqcookie
    networks:
      - backend
  vmq:
    image: vernemq/vernemq:1.12.3
    hostname: 'vmq{{.Task.Slot}}'
    environment:
      - DOCKER_VERNEMQ_SWARM=1
      - DOCKER_VERNEMQ_ACCEPT_EULA=yes
      - DOCKER_VERNEMQ_ALLOW_ANONYMOUS=on
      - DOCKER_NET_INTERFACE=eth0
      - DOCKER_VERNEMQ_LISTENER__TCP__DEFAULT=0.0.0.0:1883
      - DOCKER_VERNEMQ_DISTRIBUTED_COOKIE=vmqcookie
      - DOCKER_VERNEMQ_DISCOVERY_NODE=vmq0
    deploy:
      replicas: 1
    networks:
      - backend
networks:
  backend:
#    external: true
    name: backend
    driver: overlay
    attachable: true
    driver_opts:
      encrypted: "true"

To replicate, all you need is two virtual machines in a swarm and then deploy this compose. To check for success, use vmq1 logs. It says “unable to connect” there if it didn’t work. The issue doesn’t seem to be bound to a certain network environment as I tried two different resulting in the same error pattern.
I tried defining a gateway and subnet by hand with no effect. Also port 4000 seems to be free and unused in my networks, 3999 didn’t work either.

Did anyone else run into similar problems? Any ideas?. :frowning:

2

This was a bug fixed in libnetwork/d/overlay: support encryption on any port by corhere · Pull Request #45636 · moby/moby · GitHub; you should see it backported to currently maintained branches soon.