Docker Plugin fails when loaded to another user

  • Docker Plugin Error
  • RHEL 8.9
  • Docker 24.0.7

I’m trying to use the sbom plugin. When installed from the script, as root, it works fine. The plugin ends up in root’s home directory and generates sbom output for my specified container.

We use GitLab for CI/CD. GitLab uses ‘runners’ as an interface to build and deploy our containers into Kubernetes. The runner is simply another Linux machine that receives jobs and executes them (many of them CLI commands (e.g. sed, docker, etc.)). The user account on the runner that executes the jobs from GitLab is, ‘gitlab-runner’.

After getting the sbom plugin working as root, my intention was to use it in my CI/CD processes. And since Docker already worked running as ‘gitlab-runner’, I assumed SBOM would too. It did not.

Initially discovered that the the plugin got loaded into ~/.docker/cli-plugins/docker-sbom. So that seemed simple enough, the plugin needs to be loaded into the “gitlab-runner” account. So I su’d over to gitlab-runner, re-ran the install script (from GitHub) and the plugin installed without error. However, running ‘docker sbom’ seemed to indicate it still wasn’t loaded. Running, ‘docker’, produce output indicating there was a plugin issue:

Invalid Plugins:
sbom failed to fetch metadata: fork/exec /home/gitlab-runner/.docker/cli-plugins/docker-sbom: permission denied

I compared the owner and file permissions of ‘docker-sbom’ in both root’s and gitlab-runner’s home. The file permissions were identical and the owners were each user’s owner. Here is root’s (working):

-rwxr-xr-x. 1 root root 21413888 Nov 9 18:07 /root/.docker/cli-plugins/docker-sbom

and here is gitlab-runner’s:

-rwxr-xr-x. 1 gitlab-runner gitlab-runner 21413888 Nov 16 22:48 /home/gitlab-runner/.docker/cli-plugins/docker-sbom

What am I missing?

Please share the output of these commands:

id # as gitlab-runner
stat /var/run/docker.sock


uid=1002(gitlab-runner) gid=1002(gitlab-runner) groups=1002(gitlab-runner),979(docker) context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

stat (as gitlab-runner):

stat: cannot statx ‘/var/run/docker/sock’: Permission denied

stat (as root):

stat /var/run/docker.sock
File: /var/run/docker.sock
Size: 0 Blocks: 0 IO Block: 4096 socket
Device: 18h/24dInode: 23521 Links: 1
Access: (0660/srw-rw----) Uid: ( 0/ root) Gid: ( 979/ docker)
Context: system_u:object_r:container_var_run_t:s0
Access: 2023-11-22 01:07:33.948875051 +0000
Modify: 2023-11-18 22:12:13.404066600 +0000
Change: 2023-11-18 22:12:13.408066600 +0000
Birth: -

Since the user is in the group 979, which is allowed to access the docker.sock. So this can’t be the problem.

Though, it appears that selinux is responsible for this behavior. Since I don’t use RHEL/selinux based systems, I can not tell you how it needs to be fixed.

1 Like

Thanks. I’ll try to solve the SELinux issue and post back.

1 Like

Just got back to my computer. So based on your feedback, I tried a couple of things:

Stopped enforcement for SELinux just as a blanket, does the communication work with the “firewall” turned off sort of thing, via setenforce 0, su’d over to gitlab-runner and still getting the same error.

Also, because of the id command, I see that the ‘s’ flag is set on user bit and I believe that means it always has to run as the calling user (or something like that??) so I tried to chown root:root the docker-sbom binary in gitlab-runner’s home.

I also tried rebooting after each setting change and none of it had any effect.

Regardless, neither worked. Here is trying to run it as gitlab-runner with setenforce 0:

Am I mistaken about setenforce 0 as completely disabling any lockdowns that might be occurring from SELinux?

Edit: I forgot to mention that I grep’d /var/log/audit/audit.log (where SELinux writes its actions) and while there were ‘docker’ entries in there, they were allowed. greping for ‘docker-sbom’ and just ‘sbom’ returned nothing.

Thank you for keeping us updated!

If it even happens with selinux in permissive mode, then it must be something else.

Though, from my understanding, permissions issue should boil down to unix file permissions, selinux (or apparmor it the os uses it instead), or ACLs. We checked the first two (and you deactivated the second). So what’s left are ACLs, but I somewhat don’t believe they are the cause.

I am afraid you will have to wait for someone that actually uses a RHEL based system and knows the challenges of running Docker on RHEL.