DNS inside container not working when DNS Server is another container

I’m currently working on a little home project where i host various services on a raspberry pi 4 via docker. While working on this project i now encountered a dns problem which i can’t really get my head around. I’m hosting pihole inside a container and configured it to use my router as an upstream dns server. On my router i have configured my raspberry pi as the local dns server and added a fiew other upstream dns servers. From my understanding this would lead to all dns requests getting routed trough my pihole container on my raspberry pi and then back to my router to get it resolved. So far this setup works for all my devices on my local network including the raspberry pi itself.

The only problem i now encounter is with other containers on the same raspberry pi that are inside the same and/or different networks than pihole. All of them seem to have problems with resolving dns queries. For example: I have a phpmyadmin countainer connected to the same docker network as the pihole container. If i now ssh into the phpmyadmin container and want to execute ‘ping google.com’ or ‘apt-get update’ it won’t be able execute these commands because of failing dns.

What i already checked:

  1. I looked at /etc/resolv.conf of the phpmyadmin container => It includes 127.0.0.11 - which is correct by my knowlegde
  2. I looked at /etc/resolv.conf of the host => It includes the actual ip of my raspberry pi (NOT 127.0.0.1). I do not understand why it uses the actual ip instead of localhost here but it does work anyway
  3. I restarted docker daemon
  4. I recreated the networks included in my docker-compose.yml
  5. I recreated the phpmyadmin container

So far none of the above steps solved the problem.

Out of curiosity i then set the ip of my router in /etc/dhcpcd.conf on my host as a static nameserver and reloaded both the dhcpcd and docker daemon. If i now ssh into my phpmyadmin container dns suddenly works. I excluded my routers ip again to verify my problem and dns stops working immediately. This leads me to the conclusion that all my docker containers (excluding pihole - because i specified dns 127.0.0.1 for this container) seem to have a problem with using my hosts ip address for dns.

My current docker-compose.yml:

version: '3'

services:
  portainer:
    image: portainer/portainer-ce:linux-arm
    container_name: portainer
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
    networks:
      - frontend
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock
      - portainer_data:/data
    labels:
      - traefik.enable=true
      - traefik.docker.network=compose_frontend
      - traefik.http.routers.portainer.entrypoints=web_tcp
      - traefik.http.routers.portainer.rule=Host(`portainer.mydomain`)
      - traefik.http.services.portainer.loadbalancer.server.port=9000


  traefik:
    image: traefik:latest
    container_name: traefik
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
    networks:
      - frontend
    ports:
      - 80:80
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /home/farmadmin/config/traefik:/etc/traefik
    labels:
      - traefik.enable=true
      - traefik.docker.network=compose_frontend
      - traefik.http.routers.traefik.entrypoints=web_tcp
      - traefik.http.routers.traefik.rule=Host(`traefik.mydomain`)
      - traefik.http.services.traefik.loadbalancer.server.port=8080

  pihole:
    image: pihole/pihole:latest
    container_name: pihole
    restart: unless-stopped
    environment:
      TZ: Europe/Berlin
    networks:
      - frontend
    dns:
      - 127.0.0.1
    ports:
      - 53:53/tcp
      - 53:53/udp
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - etc-pihole:/etc/pihole/
      - etc-dnsmasq.d:/etc/dnsmasq.d/
    labels:
      - traefik.enable=true
      - traefik.docker.network=compose_frontend
      - traefik.http.routers.pihole.entrypoints=web_tcp
      - traefik.http.routers.pihole.rule=Host(`pihole.mydomain`)
      - traefik.http.routers.pihole.middlewares=dashboard_prefix
      - traefik.http.middlewares.dashboard_prefix.addprefix.prefix=/admin
      - traefik.http.services.pihole.loadbalancer.server.port=80

  mariadb:
    image: linuxserver/mariadb:latest
    container_name: mariadb
    restart: unless-stopped
    environment:
      - TZ=Europe/Berlin
      - PUID=1000
      - PGID=1000
    networks:
      - backend
    volumes:
      - mariadb_data:/config

  phpmyadmin:
    image: phpmyadmin:latest
    container_name: phpmyadmin
    restart: unless-stopped
    environment:
      - TZ=Europe/Berlin
      - PMA_HOST=mariadb
      - PMA_PORT=3306
    networks:
      - frontend
      - backend
    labels:
      - traefik.enable=true
      - traefik.docker.network=compose_frontend
      - traefik.http.routers.phpmyadmin.entrypoints=web_tcp
      - traefik.http.routers.phpmyadmin.rule=Host(`phpmyadmin.mydomain`)
      - traefik.http.services.phpmyadmin.loadbalancer.server.port=80

networks:
  frontend:
  backend:
    internal: true

volumes:
  # Persistent Portainer Data
  portainer_data:
  
  # Persistent Pihole Data
  etc-pihole:
  etc-dnsmasq.d:

  # Persistent MariaDB Data
  mariadb_data:

So my questions would be: Why does the hosts resolv.conf include its full own ip instead of localhost? Why is my host able to resolve dns queries with its own ip but my docker containers aren’t? How can i solve this problem without setting the hosts nameserver to my router?

On a sidenote: I was able to fix my issue above by setting a static ip for the pihole container and manually setting this ip as dns server for every other container.

The solution provided on DNS not working within docker containers when host uses dnsmasq and Google's DNS server are firewalled? - Stack Overflow looks a lot cleaner to me tho.

For people still coming here, i found an even easier way to solve the problem then provided in my initial answer post above: Docker container DNS not working with pihole - Unix Stackexchange