What would cause containers to stop getting resolve.conf from host system on reboot?

Docker Engine General
1

I have been running Docker on my Linux (Debian) box for years. I have multiple containers and I have never had any issues.

Recently (like the last few months), when my system reboots, all the containers start properly but they cannot access the internet. After some digging I learned its because the containers are not getting the resolve.conf from the host system.

If I restart the containers after the server has been up for a few minutes, then the containers work properly.

What would cause this? How can I debug and fix it?

2

This depends on which network a container is attached to. If a container is attached to a user defined network - created by docker network create <name> and used with docker run --network <name> ... or docker compose - then /etc/resolv.conf should point to dockers internal DNS server - without it DNS-based service discovery wouldn’t be possible.

Thus, only containers attached to the default bridge do mount the point in time version of resolv.conf. Single files or folders are not actually mounted by path, their inode is resolved and the mount happens via inode. While this works fine for directories, it can be surprising for files: if a file’s content changes, its inode changes - the container has no chance to see the change, as it only know about the inode it mounted.

Is it safe to assume that the affected containers are attached to the default bridge?

3

Just a small clarification: content can change unless a new file is created with the new content and replaces the original file. I’m not sure how that file is changed. I know that some text editors create temporary files and replaces the original file when closing the edited one. That broke bind mounts indeed.

A guess from me is that the resolv.conf does not exist yet or the content is not the same before Docker starts. I don’t remember if systemd-resolved or other similar services had to be added to docker service depencencies before

4

I have no user defined networks. All my containers are starting from compose. Some of the containers are using network_mode: host.

I just checked. Before rebooting my machine, here is the /etc/resolv.conf for one container:

# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.

nameserver 192.168.xxx.xxx
search [redacted].[redacted]

# Based on host file: '/etc/resolv.conf'
# Overrides: []

Here is the same file right after rebooting.

# Generated by Docker Engine.
# This file can be edited; Docker Engine will not make further changes once it
# has been modified.


# Based on host file: '/etc/resolv.conf'
# Overrides: []
5

Indeed, using the network_mode: host does not use a container network at all. It is the absence of network isolation from the hosts network namespace.

We can put the inode idea to a test. Run this command inside the container when it after a reboot and after restarting the contaienr :

ls -i /etc/resolv.conf

It will print the inode number. If it really mounts the inode from the host, the number should be identical inside the container and outside on the host. If the file is updated inplace, the inode number should be identical for both containers.

Is it possible that a service that rewrites the resolv.conf is started after the docker service is started?

6

I figured it out. Rather, Claude did.

I had to change allow-hotplug enp2s0 in /etc/network/interfaces to auto enp2s0 and it worked.

allow-hotplug let your NIC come up in the background without boot waiting for it, so Docker started containers before DHCP had even finished — no DNS yet. auto makes boot wait for the NIC to fully get its IP and DNS first, so by the time Docker starts, real DNS is already there.

7

I am glad you were able to sort it out!

Better solution than what I had in mind: detect which service does it and modify the docker systemd unit to add the service that is responsible to the “After” list.