I have been running all my containers using a Macvlan network type which has simplified all my firewall (external) rules as I can identify each docker container by IP address. (I can also create an ALIAS name for the IP address in my firewall to identify the container easier.).
I was initializing all my containers with docker run commands and am now converting everything to docker-compose files. In the process I am wondering, is running a macvlan network is the best practice?
Or would it be more common to create a custom bridge network and configure my firewall rules to access to containers using ports? The Letsencrypt container will need to reach some containers but not all.
There is no general answer to this, as each environment is different.
Though, what I can tell is that I was running docker in enterprise environments for years without requiring or using macvlan at all. My experience is that usualy either people comming from a vm background or home users feel they absolutly need macvlan.
I always ask myself what my objective is, what my options are, and what provides the cleanest solution with the least effort.
Personaly, I prefere a reverse proxy like Traefik as entrypoint for all incomming https over macvlan any time. I am neither keen to learn ips nor ports of my applications. I just want to access my services without having to memorize useless stuff.
For instance one of the feature of Traefik are domain based or path based reverse proxy rules, attached as container labels to the target service, which get added/removed in traffic whenever the service is started/stopped.
How I use it in my homelab:
I have several public domains, which point a *. subdomain to my WAN ip. I forward port 443 to my Swarm Cluster where Traefik listens on port 443. Traefik keeps track on managing wildcard letsencrypt certificates for my domains and forwarding the traffic to the right backend container. I just have to add new subdomains to my container labels and that’s all I need to make another container accessible for the outside world.
Of course this could be done purely internal with a dns server you run in your network, like unbound or pihole - both allow to create entries for individual domain names.
I have setup my 3 different stacks on 3 different networks and everything works. However I previously had settings in my firewall (separate devices) that allowed me to filter traffic to go out through a VPN if it came from a specific IP address, now I can no longer do that.
As docker networks basicly provide SNAT there is no way to distinguish the containers.
You could introduce a forward proxy and configure it in the containers that are supposed to use the vpn and then alllow the forward proxy in your firewall, though it feels less cleaner then actualy using macvlan. Of course the forward proxy would only make sense where the protocol supports it, like http,ftp or socks in general.
N.B.: it is possible to run a vpn client containers and hook other containers into the network namespace of the vpn client container to make them use the vpn connection.