Permissions issues on /etc/hosts and resolv.conf for new containers

Docker Engine General
1

I’m running Docker 26.1.3 on AlmaLinux which is also running WHM w/Cpanel.

I’ve been running Docker for monitoring containers, Prometheus, Loki, and Grafana. Honestly, a fairly generic stack. It’s been working fine with no issues.

Recently I needed to rebuild my Prometheus container and it came up with lots of network problems. I tracked it to DNS issues inside the new container and more specifically, the default container user (Nobody) wasn’t able to read:

  • /etc/hosts (640)
  • /etc/resolv.conf (640)

This of course is inside the contaner. The default for Prometheus is user NOBODY.

I was able to fix the issue by just using chmod 644 on the files but, this is a pain the I’m just trying to figure out what the issue is.

Doing more testing, if I build just a test container:

[root@host etc]# docker run --rm -it alpine:latest ls -l /etc/resolv.conf /etc/hosts
-rw-r-----    1 root     root           174 Dec 15 05:34 /etc/hosts
-rw-r-----    1 root     root           265 Dec 15 05:34 /etc/resolv.conf

On all my other hosts, the files are correctly permissions with 644.

Anyone have any ideas on what I can try? Thank you in advance!

Server: Docker Engine - Community
Engine:
Version: 26.1.3
API version: 1.45 (minimum version 1.24)
Go version: go1.21.10
Git commit: 8e96db1
Built: Thu May 16 08:33:34 2024
OS/Arch: linux/amd64
Experimental: false
containerd:
Version: 1.6.32
GitCommit: 8b3b7ca2e5ce38e8f31a34f35b2b68ceb8470d89
runc:
Version: 1.1.12
GitCommit: v1.1.12-0-g51d5e94
docker-init:
Version: 0.19.0
GitCommit: de40ad0

2

This is not the default behavior so I assume it is changed by SELinux. Note that AlaLinux is not officially supported by Docker. Even if AlmaLinux uses CentOS repositories, currently the only supported CentOS is CentOS 9 stream and using a repository is not a guarantee of being compatible.

On the other hand, a similar issue happened on CentOS as well before:

It was different as the file was not even readable by root.

If nobody knows the exact, correct solution here, I can recommend an AlmaLinux community.

The resolv.conf file by the way is not strictly part of the container filesystem. At least not part of the image. It is a copy of the resolv.conf on your host and mounted from the host. You can check the permissions on the host this way as root:

ls -la $(docker container inspect CONTAINERNAME --format '{{ .ResolvConfPath  }}')

and for the hosts file

ls -la $(docker container inspect CONTAINERNAME --format '{{ .HostsPath  }}')

Unfortunately I never installed Docker on AlmaLinux and I never had this issue on CentOS when I used it, so I’m not sure what could change the permission if it was not SELinux somehow. It could be any policy that doesn’t allow files under an assumed system folder to be readable by a non-root user.