When you initialize a swarm or join a Docker host to an existing swarm, an overlay network called ingress is created on that Docker host, which handles the control and data traffic related to swarm services.
Docker Swarm will automatically create an overlay network, which does not work root-less.
As I understand it. And as I have read it in various places:
Swarm mode creates an overlay network on the Docker host that handles control and data traffic related to the swarm services. It also uses virtual IP addresses (VIP), virtual extensible LAN (VXLAN) and modifies iptables. For this purpose, the Docker Deamon must have root privileges.
It’s just a guess, but it probably need to replace more than just the overlaying or underlaying network. And it probably wouldn’t change the fact that the deamon needs root privileges. From what I’ve read and understood, it’s probably more of a Linux thing than a Docker thing.
I can only guess that it will need extensive knowledge of Docker, Linux, virtual networks and possibly even programming skills in GO to implement that suggestion. And yet I would be unsure whether it can work without root privileges.
Instead of rootless mode, some places I’ve read have recommended using the Linux user namespace.
I think you can tell Swarm to use wireguard by appending --data-path-addr wg0 to your docker swarm init and docker swarm join commands. It still overlays your wireguard network, but at least it will prevent traffic from going unecrypted over untrusted networks.