The embedded dns server not refresh the host machine dns in some cases

General
1

Centos 7.9 , Docker version 26.1.4

Summarize:
The embedded dns server not use the new dns for the old container when the host machine dns has changed successfully, but it work for new creating contianer.

Detail:
I created my network “www”, it is bridge network, just be configed the “Subnet”: “172.88.0.0/24” and “Gateway”: “172.88.0.1”.

Then I simple create a container A1 with the config --network=www and specify an IPV4 which in the www subnet such as 172.88.0.12. The nameserver in the /etc/resolv.conf in the container A1 only have one line and it is nameserver 172.0.0.11.

I know this is the docker embedded dns, and now the network everything is normal OK, such as I try the nslookup www.google.com and get the same result as I try in Host Machine. Now the Host Machine dns(nameserver in /etc/resolv.conf in Host Machine ) is 192.168.22.1.

And then, I change the Host Machine network environment(such as I take my notebook from company to home), the Host Machine dns will change to 192.168.1.1. The nslookup www.google.com is OK in the Host Machine but not OK in container A1 (the container A1 and the dockerd service no restart).

I try to analyze the promble. I run the nslookup www.google.com in container A1 and I use tcpdump watch the container A1 net by nsenter --net={container_A1_nets_link} tcpdump -i any -nn in the Host Machine, I see that

IP 127.0.0.1.54976 > 127.0.0.11.52502: UDP, length 31
IP 127.0.0.1.54976 > 127.0.0.11.52502: UDP, length 31
IP 172.88.0.2.36647 > 192.168.22.1.53: 22594+ A? www.google.com. (31)
IP 172.88.0.2.38885 > 192.168.22.1.53: 22893+ AAAA? www.google.com. (31)
......

Obviously, it is still using the old dns address 192.168.22.1 now ! It sholud be the new DNS 192.168.1.1 .

At this moment I try these:

  1. ping 192.168.1.1 in container A1, it is OK;
  2. ping 192.168.22.1 in container A1, it is not OK;
  3. nslookup www.google.com 192.168.1.1 in container A1, it is OK;

And then, I simple create a container B1 as same as container A1 with another subnet IPV4 in the www network, and do the same analyze.

The container B1 work normal OK everything. The nslookup and the ping command is OK, the tcpdump show using the new DNS 192.168.1.1.

So how let the container A1 work OK? I found it will work normal OK everything after I restart it !

Doubt:
In this using cases, why the embedded dns server 127.0.0.11 not use the new DNS? Is the embedded dns server in the every container so that it must be “refresh” the new DNS by restart container or cretea the new one?
I don’t want to restart my all conainer in each network environment change. The calude suggest me to set a myselft dns server to replace the embedded dns server, maybe it will work out but it mean I should give up using the embedded dns server, but something I need it such as I connet with container name in the network www subnet.
So it is the embedded dns server design bug and should be fixed?

2

Please update to current OS and Docker, it could be an issue that has long been fixed.

CentOS Linux 7 reached end of life (EOL) on June 30, 2024.

3

I’m not sure I completely understood the issue, but I wrote about my test results on how Docker uses external DNS servers and when those server addresses are updated

So even if I created a new container, it didn’t use the new DNS until I restarted the Docker daemon. Even if the Docker Daemon already knows about the new DNS servers, Docker creates a resolv.conf for each container and copies it under /var/lib/docker/containers among other files like the hosts file. to mount to the container. So I wouldn’t expect it to be recreated just because the container restarts either, but I would when the container is recreated since that would create a completely new container with a new resolv.conf.

If you are wondering why it works like that and why it doesn’t immediately updates the DNS servers for containers, I guess because this way containers are less likely to be affected by something on the host since the container is an isolated environment which has its own filesystem including the DNS. Docker just helps you to automatically create a resolv.conf so you don’t have to put it into the container. But Docker also supports resolving compose service names or container names to container IP addresses so that is why it uses 127.0.0.11 but it is not running in the containers directly, although I’m not entirely sure about the exact implementation. But you still have the dns config that was created for the container, it just also allows you to resolve internal domain names.

4

Before I asked the question, I had read your article, but I found that it was a bit different from what I am currently using.

For example, as I mentioned earlier, when the host machine is converted to a new DNS and a new container is created using the embedded DNS server (127.0.0.11), nslookup in the container can be used to observe the use of the new DNS. same as a new container is created through the host network. There is no need to restart the Docker service.

I can understand that for a created container, the reslov.conf file inside it is created independently. If this file needs to be updated, the Docker service should be restarted, but in reality, restarting the container is enough.

If it is simply to let the container directly use the DNS of the host machine, I can directly mount the reslov.conf of the host machine into the container. However, in reality, I need to use Docker’s embedded DNS server, which is completely impossible for users to maintain or restart separately.

it would be best to have any “operation” to update the upstream service address of this embedded DNS server !

My temporary solution is:
To establish a dnsmasq container service for the host network and map ports to the host. All bridge type network containers will have an additional DNS (using gateway IP) added during creation. At this point, the DNS server of the container is still using the embedded DNS server, but the upstream service address of the embedded DNS server is the dnsmasq container service. Then, I can achieve “timely updates” by managing the dnsmasq container service.

Although this solution is somewhat similar to CoreDNS in Kubernetes, if Docker’s embedded DNS server can update the upstream service address to the new DNS address of the host machine in a timely manner, users generally do not need this solution to solve problems in such scenarios.