Docker on Fedora 27 Server Edition: SELinux shenanigans resulted in all containers getting access denied on start

Hi,

I recently installed Docker to my Fedora 27 Server.

Issue type

Container File System Permission

OS Version/build

Linux fed27 4.16.5-200.fc27.x86_64 #1 x86_64

App version

$ sudo docker version
Client:
Version: 1.13.1
API version: 1.26
Package version: docker-1.13.1-51.git4032bd5.fc27.x86_64
Go version: go1.9.4
Git commit: 7f1fa5c-unsupported
Built: Wed Mar 28 13:58:10 2018
OS/Arch: linux/amd64

Server:
Version: 1.13.1
API version: 1.26 (minimum version 1.12)
Package version: docker-1.13.1-51.git4032bd5.fc27.x86_64
Go version: go1.9.4
Git commit: 7f1fa5c-unsupported
Built: Wed Mar 28 13:58:10 2018
OS/Arch: linux/amd64
Experimental: false

Steps to reproduce

  1. Install Docker.
  2. Create volumes, networks, then containers using those.
  3. Uninstall Docker Without stopping or removing the containers
  4. rm -rf the docker directory
  5. unmount the overlayfs and volume directories
  6. rm -rf the rest of the docker directory
  7. Reinstall Docker
  8. Create new containers via docker run

I realize I messed up, but now I can’t get the containers for docker to work correctly.

If I setenforce 0, the containers run fine. That’s how I know it should be an SELinux issue.

Doing sudo restorecon -Rv / did not solve the issue. The results of doing that was the various container files in the docker folder being set to container file system files. I don’t have the actual results anymore or I’d post them explicitly.

The commands used to produce the containers:

sudo docker run -dit --name ether --network backend -p 25000:80 -v ether-store:/var/www/html/:z -d php:7.0-apache
sudo docker run --name stone --network backend -v stone-store:/var/lib/mysql:z -e "MYSQL_ROOT_PASSWORD=You beckon me to the Cross. " -d mysql:latest

What the logs say:

$ sudo docker logs ether
standard_init_linux.go:178: exec user process caused “permission denied”
$ sudo docker logs stone
standard_init_linux.go:178: exec user process caused “permission denied”

$ sudo systemctl status docker
● docker.service - Docker Application Container Engine
Loaded: loaded (/usr/lib/systemd/system/docker.service; disabled; vendor preset: disabled)
Active: active (running) since Sat 2018-05-05 19:51:57 CDT; 33min ago
Docs: http://docs.docker.com
Main PID: 4515 (dockerd-current)
Tasks: 18 (limit: 8192)
Memory: 484.7M
CPU: 36.381s
CGroup: /system.slice/docker.service
└─4515 /usr/bin/dockerd-current --add-runtime oci=/usr/libexec/docker/docker-runc-current --default-runtime=oci --authorization-plugin=rhel-push-plugin --containerd /run/containerd.sock --exec-opt native.cgroupdriver=systemd --userland-proxy-path=/usr/libexec/docker/docker-proxy-current --init-path=/usr/l

May 05 19:57:43 fed27 dockerd-current[4515]: time=“2018-05-05T19:57:43.592795262-05:00” level=error msg=“Handler for POST /v1.26/containers/create?name=stone returned error: No such image: mysql:latest”
May 05 19:57:43 fed27 dockerd-current[4515]: time=“2018-05-05T19:57:43.593624089-05:00” level=error msg=“Handler for POST /v1.26/containers/create returned error: No such image: mysql:latest”
May 05 19:58:17 fed27 dockerd-current[4515]: time=“2018-05-05T19:58:17-05:00” level=info msg=“Firewalld running: true”
May 05 19:58:18 fed27 dockerd-current[4515]: standard_init_linux.go:178: exec user process caused “permission denied”
May 05 19:58:19 fed27 dockerd-current[4515]: time=“2018-05-05T19:58:19.496139950-05:00” level=warning msg=“4f974538eb0fe15f0f626b3d43f0b9195b75e4bc815cbe23f937f69f73abf674 cleanup: failed to unmount secrets: invalid argument”
May 05 19:58:47 fed27 dockerd-current[4515]: time=“2018-05-05T19:58:47.141993137-05:00” level=error msg=“Handler for POST /v1.26/containers/create?name=ether returned error: No such image: php:7.0-apache”
May 05 19:58:47 fed27 dockerd-current[4515]: time=“2018-05-05T19:58:47.142837045-05:00” level=error msg=“Handler for POST /v1.26/containers/create returned error: No such image: php:7.0-apache”
May 05 19:59:40 fed27 dockerd-current[4515]: time=“2018-05-05T19:59:40-05:00” level=info msg=“Firewalld running: true”
May 05 19:59:41 fed27 dockerd-current[4515]: [73B blob data]
May 05 19:59:43 fed27 dockerd-current[4515]: time=“2018-05-05T19:59:43.134642894-05:00” level=warning msg=“031ff7fdab9aea2e3a81668854b81053a74ef686acc50a28d3018195852e1762 cleanup: failed to unmount secrets: invalid argument”

I’m not really sure where to go from here.

Edit:

I get an alert that source process: runc:[2:INT] attempted to access: entrypoint On this file: docker-php-entrypoint.

It advises that I do this:

/sbin/restorecon -v /usr/local/bin/docker-php-entrypoint

However, that file doesn’t exist on the host system. It only exists within the container’s overlayfs. I checked this with sudo find / -type f -name docker-php-entrypoint.sh and this was the only result: /var/lib/docker/overlay2/4e3578e28f357e884eeaf654ea503973ddaa4fdb5df84e2b923e2fa9ca2fac34/diff/usr/local/bin/docker-entrypoint.sh

Doing restorecon -v on the overlayfs file doesn’t do anything either.

It seems like Overlayfs is being prevented from redirecting to the appropriate container file system and is instead trying to run on the host itself (which SELinux is stopping as it should).

Disabling SELinux enforcement allowing the redirection seems weird to me though in why it would happen.

We have an issue close to yours. In the /var/lib/docker/overlay/xxxxxx/ directories.
On one server, the content of the container has the right SElinux configuration (system_u:object_r:container_file_t:s0) and it works.
And the second one, we have (system_u:object_r:container_share_t:s0) so the container can’t write on his own filesystem.
We don’t know yet what we did differently, we are reinstalling docker and may be the system to see how it goes.
I will update soon.
[update]

So, I tried to reinstall the server, same OS version (Centos 7.5) on FPV, same issue. The only difference between the working and non working is that the working one is a physical server.