DNS lookups via internal pi-hole container failing after restart

I’ve been running several compose apps on the same desktop/lxc in my LAN for several months now. One of them is a pi-hole container that serves DNS lookups for the entire LAN. After attempting to set up endlessh yesterday and restarting the host, pi-hole queries no longer resolve from containers, but still do resolve properly from other computers on the network.

checking the output of the docker service shows:

Jan 10 13:54:25 docker dockerd[74]: time="2025-01-10T13:54:25.729965601-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:37796" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:37796->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.\tIN\t AAAA"
Jan 10 13:54:25 docker dockerd[74]: time="2025-01-10T13:54:25.730078622-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:49015" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:49015->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.\tIN\t A"
Jan 10 13:54:29 docker dockerd[74]: time="2025-01-10T13:54:29.029520216-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.30.0.2:49237" dns-server="udp:192.168
.13.78:53" error="read udp 172.30.0.2:49237->192.168.13.78:53: i/o timeout" question=";cloud.csb.sh.local.\tIN\t A"
Jan 10 13:54:29 docker dockerd[74]: time="2025-01-10T13:54:29.731811801-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:46127" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:46127->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.\tIN\t AAAA"
Jan 10 13:54:29 docker dockerd[74]: time="2025-01-10T13:54:29.731903667-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:55750" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:55750->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.\tIN\t A"
Jan 10 13:54:33 docker dockerd[74]: time="2025-01-10T13:54:33.030587417-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.30.0.2:33325" dns-server="udp:192.168
.13.78:53" error="read udp 172.30.0.2:33325->192.168.13.78:53: i/o timeout" question=";cloud.csb.sh.local.\tIN\t A"
Jan 10 13:54:33 docker dockerd[74]: time="2025-01-10T13:54:33.733693909-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:34176" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:34176->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.local.\tIN\t A"
Jan 10 13:54:33 docker dockerd[74]: time="2025-01-10T13:54:33.733766494-06:00" level=error msg="[resolver] failed to query external DNS server" client-addr="udp:172.28.0.6:45071" dns-server="udp:192.168
.13.78:53" error="read udp 172.28.0.6:45071->192.168.13.78:53: i/o timeout" question=";time.cloudflare.com.local.\tIN\t AAAA"

The pi-hole container is up, and bound to port 53.

root@docker:~# docker ps | grep hole
510cbe4431f4   pihole/pihole:latest                           "/s6-init"               About an hour ago   Up 52 minutes (healthy)           0.0.0.0:53->53/tcp, 0.0.0.0:8060->8060/tcp, 0.0.0.0:53->53/udp, 67/udp, 80/tcp   pihole

Querying from the docker host works:

root@docker:~# dig google.com

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 55956
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             25      IN      A       172.253.124.100
google.com.             25      IN      A       172.253.124.138
google.com.             25      IN      A       172.253.124.102
google.com.             25      IN      A       172.253.124.139
google.com.             25      IN      A       172.253.124.113
google.com.             25      IN      A       172.253.124.101

;; Query time: 8 msec
;; SERVER: 192.168.13.78#53(192.168.13.78) (UDP)
;; WHEN: Fri Jan 10 14:18:07 CST 2025
;; MSG SIZE  rcvd: 135

Querying from inside a container does not:

 5cd032d7aed6 _ ~ _ dig google.com
;; communications error to 192.168.13.78#53: timed out
;; communications error to 192.168.13.78#53: timed out
^C#                                                                                                                                                                                                       

 5cd032d7aed6 _ ~ _ dig google.com @127.0.0.11
;; communications error to 127.0.0.11#53: connection refused
;; communications error to 127.0.0.11#53: connection refused
;; communications error to 127.0.0.11#53: connection refused

; <<>> DiG 9.18.25 <<>> google.com @127.0.0.11
;; global options: +cmd
;; no servers could be reached


 5cd032d7aed6 _ ~ _ dig google.com @8.8.8.8   

; <<>> DiG 9.18.25 <<>> google.com @8.8.8.8
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 28919
;; flags: qr rd ra; QUERY: 1, ANSWER: 6, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             252     IN      A       108.177.122.138
google.com.             252     IN      A       108.177.122.100
google.com.             252     IN      A       108.177.122.139
google.com.             252     IN      A       108.177.122.101
google.com.             252     IN      A       108.177.122.102
google.com.             252     IN      A       108.177.122.113

;; Query time: 8 msec
;; SERVER: 8.8.8.8#53(8.8.8.8) (UDP)
;; WHEN: Fri Jan 10 20:19:21 UTC 2025
;; MSG SIZE  rcvd: 135

The pi-hole compose:

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    #network_mode: host
    ports:
      - 53:53/tcp
      - 53:53/udp
      - 8060:8060/tcp
    environment:
      TZ: America/Chicago
      WEBPASSWORD: ...
      WEB_PORT: 8060
    # Volumes store your data between container upgrades
    volumes:
      - /mnt/data/pi-hole/etc:/etc/pihole
      - /mnt/data/pi-hole/dnsmasq-d:/etc/dnsmasq.d
    restart: unless-stopped
networks: {}

I’m not sure how to troubleshooting this, and would appreciate any pointers.

It seems like a networking issue.

192.168.13.78#53 is the IP of the Docker host?

You restarted, so some IPs might have changed.

Any software and/or firewall updates done?

Thank you for the troubleshooting pointers.

Yes, 192.168.13.78 is the static docker host, and port 53 tcp/udp is forwarded to a pihole docker container. It is reachable by everything else in the LAN except from inside other docker containers, which are able to reach other exposed container ports on the same host IP.

I did upgrade the Debian packages, and it took docker from docker-ce-cli:amd64 5:27.3.1-1~debian.12~bookworm 5:27.4.1-1~debian.12~bookworm to docker-ce-cli:amd64 5:27.4.1-1~debian.12~bookworm <none>. Is there a chance that updated the docker iptable rules?

Removing the exposed ports and switching to network_mode host works, but feels like a cop-out to a real solution.

services:
  pihole:
    container_name: pihole
    image: pihole/pihole:latest
    # For DHCP it is recommended to remove these ports and instead add: network_mode: "host"
    network_mode: host
....