Issues Deploying Pi-hole with Macvlan and Tailscale on Docker Swarm

jps229 · September 2, 2024, 12:57pm

Hi.
I’m currently facing challenges with deploying a second Pi-hole instance within my Docker Swarm environment. This Pi-hole is intended to serve as a backup to a physical Pi-hole already in operation. My goal is to have this backup Pi-hole accessible with a fixed IP address on my LAN and also be reachable via Tailscale. However, despite multiple attempts, I’m encountering persistent issues, particularly with IP assignment via Macvlan and proper integration with Tailscale.

Setup Overview:

Environment: Docker Swarm running on a cluster of Raspberry Pi nodes (network-booted), managed via Portainer. I have both Raspberry Pi nodes and AMD64 nodes in my Swarm, but I want the Pi-hole to be deployed only on the Raspberry Pi nodes (all of them use eth0 as network).
Networking: I’m using a Macvlan network to assign a fixed IP within my LAN. The goal is for this service to be consistent and accessible both locally and also via Tailscale.

Docker Compose File: Here’s the current version of my Docker Compose file (Portainer stack)

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    ports:
      - "53:53/tcp"
      - "53:53/udp"
      - "8089:80/tcp"
      - "8443:443/tcp"
    environment:
      TZ: "Europe/Zurich"
      WEBPASSWORD: "********"
      DNSMASQ_LISTENING: "all"
      DNSMASQ_USER: "root"
      PIHOLE_UID: "1000"
      PIHOLE_GID: "1000"
      WEB_UID: "1001"
      WEB_GID: "1001"
    volumes:
       - "/volumesOMV/pi_hole/etc-pihole:/etc/pihole"
       - "/volumesOMV/pi_hole/etc-dnsmasq.d:/etc/dnsmasq.d"
    restart: unless-stopped
    networks:
      mvlan_live_piCluster04:
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.eth0netz == true
          - node.hostname == piCluster04

#tailscale
  tailscale_pihole:
    image: tailscale/tailscale:latest
    container_name: tailscale_pihole
    privileged: true
    hostname: dockerswarm_pihole
    environment:
      - TS_AUTH_KEY=tskey-client-*******
      - TS_EXTRA_ARGS=--advertise-tags=tag:container
      - TS_STATE_DIR=/var/lib/tailscale
    volumes:
      - /volumesOMV/pi_hole/tailscale:/var/lib/tailscale
      - /dev/net/tun:/dev/net/tun
    command: tailscaled --state=/var/lib/tailscale/tailscaled.state
    network_mode: host
    cap_add:
      - net_admin
      - sys_module
    restart: unless-stopped
    deploy:
      mode: replicated
      replicas: 1
      placement:
        constraints:
          - node.labels.eth0netz == true
          - node.hostname == piCluster04

networks:
  mvlan_live_piCluster04:
    external: true

What I’ve Tried and Considered:

Macvlan Configuration: I initially set up a Macvlan network and tried assigning a fixed IP via docker-compose. However, the container repeatedly received a different IP than specified.
Router DHCP Config: I attempted to bypass Docker’s IP assignment by setting a static IP in the router based on the container’s MAC address, but this also failed to assign the correct IP.
Single IP Range: I managed to get the container to receive the correct IP address by configuring the Macvlan network with a subnet mask of /32, effectively limiting the network to a single IP address (192.168.1.254). However, this is a workaround, and the problem persists when I try to use a broader range (e.g., /29).
Tailscale Integration: The Tailscale setup doesn’t always register automatically. Even when it does, I cannot access the Pi-hole web interface through the Tailnet IP (neither on port 80 nor 8089).

pi-hole-logs from Docker

     
s6-rc: info: service s6rc-oneshot-runner: starting
s6-rc: info: service s6rc-oneshot-runner successfully started
s6-rc: info: service fix-attrs: starting
s6-rc: info: service fix-attrs successfully started
s6-rc: info: service legacy-cont-init: starting
s6-rc: info: service legacy-cont-init successfully started
s6-rc: info: service cron: starting
s6-rc: info: service cron successfully started
s6-rc: info: service _uid-gid-changer: starting
  [i] Changing ID for user: www-data (33 => 1001)
  [i] Changing ID for group: www-data (33 => 1001)
  [i] Changing ID for user: pihole (999 => 1000)
s6-rc: info: service _uid-gid-changer successfully started
s6-rc: info: service _startup: starting
  [i] Starting docker specific checks & setup for docker pihole/pihole
  [i] Setting capabilities on pihole-FTL where possible
  [i] Applying the following caps to pihole-FTL:
        * CAP_CHOWN
        * CAP_NET_BIND_SERVICE
        * CAP_NET_RAW
  [i] Ensuring basic configuration by re-running select functions from basic-install.sh
  [i] Installing configs from /etc/.pihole...
  [i] Existing dnsmasq.conf found... it is not a Pi-hole file, leaving alone!
  [i] Installing /etc/dnsmasq.d/01-pihole.conf...
  [✓] Installed /etc/dnsmasq.d/01-pihole.conf
  [i] Installing /etc/.pihole/advanced/06-rfc6761.conf...
  [✓] Installed /etc/dnsmasq.d/06-rfc6761.conf
  [i] Installing latest logrotate script...
	[i] Existing logrotate file found. No changes made.
  [i] Assigning password defined by Environment Variable
  [✓] New password set
  [i] Added ENV to php:
                    "TZ" => "Europe/Zurich",
                    "PIHOLE_DOCKER_TAG" => "",
                    "PHP_ERROR_LOG" => "/var/log/lighttpd/error-pihole.log",
                    "CORS_HOSTS" => "",
                    "VIRTUAL_HOST" => "317f0dd26850",
  [i] Using IPv4 and IPv6
  [i] Installing latest Cron script...
  [✓] Installing latest Cron script
  [i] Preexisting ad list /etc/pihole/adlists.list detected (exiting setup_blocklists early)
  [i] Existing DNS servers detected in setupVars.conf. Leaving them alone
  [i] Applying pihole-FTL.conf setting LOCAL_IPV4=0.0.0.0
  [i] FTL binding to default interface: eth0
  [i] Enabling Query Logging
  [i] Testing lighttpd config: Syntax OK
  [i] All config checks passed, cleared for startup ...
  [i] Docker start setup complete
  [i] pihole-FTL (no-daemon) will be started as root
s6-rc: info: service _startup successfully started
s6-rc: info: service pihole-FTL: starting
s6-rc: info: service pihole-FTL successfully started
s6-rc: info: service lighttpd: starting
s6-rc: info: service lighttpd successfully started
s6-rc: info: service _postFTL: starting
s6-rc: info: service _postFTL successfully started
s6-rc: info: service legacy-services: starting
  Checking if custom gravity.db is set in /etc/pihole/pihole-FTL.conf
s6-rc: info: service legacy-services successfully started
  [✗] DNS resolution is currently unavailable

References and Tutorials Used:

Tailscale Documentation: I followed Tailscale’s Docker guide for setting up Tailscale within Docker, ensuring proper authentication and tag management.
Macvlan Configuration: I referenced several Docker and networking guides to ensure the Macvlan setup was correct, but the IP assignment remains inconsistent.

Current Status: Despite these efforts, the Pi-hole container either receives the wrong IP or experiences network conflicts that prevent it from functioning correctly. I’m reaching out for advice or suggestions from anyone who has successfully deployed a similar setup or encountered similar issues in docker swarm (as I’ve seen some people have working setups in stand-alone-docker-hosts).

Any help or insights would be greatly appreciated!

meyay · September 2, 2024, 5:12pm

Its because this feature does not exist for swarm services.

This is because docker itself takes care of ip address management, and assigns the ip itself. There is no way around it.

yup, see answers above. It even gets better: you can only have a single docker network for the same gateway ip. So creating multiple “/32” networks will not work.

I can’t say anything about tailscale.

Just to be sure: you created a --config-only network on each node (all use the same parent network interface name), then create the swarm scoped macvlan network, right? Here is an example: Using docker Swarm with Macvlan. Of course, the problem with not assignable static ips remain.

jps229 · September 2, 2024, 7:59pm

Hi meyay,

Thank you for your input. Yes, I initially created the network configuration and then set up the macvlan on the specific node. For testing purposes, I applied it only to this node and not across all nodes. Only the Raspberry Pis in my swarm are using eth0, which is why I constrained the deployment using (node.labels.eth0netz=true).

I’m also surprised to see that I currently have two macvlan networks working simultaneously, with my router acting as the gateway. These two containers, one of which is not a Pi-hole, are running on different nodes using different macvlan networks.

[piCluster04] jps229@piCluster04:~$ docker network ls | grep macvlan
h12l3btphk27   main_macvlan_jdown                  macvlan   swarm
rimoq3kqjo2n   mvlan_live_piCluster04              macvlan   swarm

So, there seem to be some exceptions here?!
Is it because I don’t spread the macvlan to all nodes? As I have 9 nodes, I would be able to build “groups”, if that is a solution.

If using macvlan with Swarm is not the ideal solution, what would you recommend as the best approach to achieve my goal? Specifically, I’m looking to have a redundant Pi-hole setup that is accessible from both the LAN and via Tailscale. Are there any alternative solutions you suggest?

meyay · September 2, 2024, 8:16pm

I am not sure why it worked for you. But it shouldn’t be possible… yet, here we are

Have you considered to use keepalived to introduce a failover ip for the lan side? Since I have no knowledge or experience with Tailscale and what it needs, I will leave it for someone else to address it.

jps229 · September 2, 2024, 8:39pm

If I would use keepalived (never done it before), it seems there’s a risk that the VIP might not align with the node where Docker Swarm has moved the Pi-hole container after a failure. One solution could be to restrict the Pi-hole service to two specific nodes and run keepalived only on those nodes to ensure they stay in sync. Do you see any alternative approach, or is this the best solution?

I’m working on network-booted nodes. Are there any known issues or special considerations I should be aware of with this setup?

meyay · September 3, 2024, 6:43pm

Personally, I use keepalived on nodes that run traefik in global mode (which can be combined with placement constraints like node labels to restrict on which nodes the tasks are deployed). If I am not mistaken, global mode services should still use the ingress routing mesh for published ports, unless you publish them in host mode.

I have no idea. Though, why would you expect it to behave differently on the network layer?

jps229 · September 8, 2024, 5:51pm

my second pi-hole is now on an unused pi400-board…

but I’ve been thinking about how to work around the issues with macvlan in Docker Swarm. Since macvlan itself works in Swarm but the IP assignment can be somewhat unpredictable across nodes, could this idea work?

What if I place all services that require macvlan into a single stack/compose file and create one macvlan network with enough IP addresses for all of them? By making the services dependent on each other using depends_on or another mechanism (delay, healthchecks), I could control the order of service startup, and possibly influence how IP addresses are assigned within the macvlan range.

Would this help ensure that IP addresses are allocated in a predictable way?

meyay · September 8, 2024, 6:36pm

You can give it a try. Personally, I would not advise depending on non-deterministic behavior. Also, depends_on is not available for swarm services. It is exclusive for services in docker compose project deployments.

If fixed ips are more important than deploying the services as swarm stack, it might be a feasable solution to deploy the services via docker compose on the nodes directly. You should be able to attach a compose service to a swarm scoped macvlan network (if the network is created with the --attachable option), and hopefully it even allows assigning a static ip address, like it does on a local scoped macvlan network.

I am aware that you want to perform a swarm stack deployments, instead of individual compose services deployed on dedicated nodes.