Docker buildx builder not visible for all users on shared vps

General Discussions
1

Hi,

We are running Jenkins as a shared service for multiple development teams and we want to add support for multiarch builds using docker buildx builders. However when trying to use a named builder from a Jenkins job it cannot be found.

We have used the following command running as the jenkins user to create the builder:

docker buildx create --name jenkins-multiarch --platform "linux/amd64,linux/amd64/v2,linux/amd64/v3,linux/amd64/v4,linux/arm64,linux/arm/v6,linux/arm/v7" --bootstrap --node localhost

When ssh:ing into the server and running docker buildx ls the output is different depending on the user issuing the command, eg:

normal user:

[user@jenkins-host ~]$ docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
default * docker
  default default         running v0.12.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

root-user via sudo:

[user@jenkins-host ~]$ sudo docker buildx ls
NAME/NODE DRIVER/ENDPOINT STATUS  BUILDKIT PLATFORMS
default * docker
  default default         running v0.12.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

jenkins-user via sudo:

[user@jenkins-host ~]$ sudo -u jenkins-local-user docker buildx ls
NAME/NODE         DRIVER/ENDPOINT             STATUS  BUILDKIT PLATFORMS
jenkins-multiarch docker-container
  localhost       unix:///var/run/docker.sock running v0.15.2  linux/amd64*, linux/amd64/v2*, linux/amd64/v3*, linux/amd64/v4*, linux/arm64*, linux/arm/v6*, linux/arm/v7*, linux/386
default *         docker
  default         default                     running v0.12.5  linux/amd64, linux/amd64/v2, linux/amd64/v3, linux/386

The strange thing is that when sudo:ing to the jenkins user we can see the bulder but the jenkins agent is not able to see the builder when running the actual job ?

Can anyone explain if there is some per-user restrictions by default or if we have done something wrong/weird filesystem permissions ?

Br.
Andreas

2

Hi,

I think I found the answer to my own question. The buildx configuration is stored in ~/.docker/buildx where builder instances are also stored.

[jenkins-local-user@jenkins user]$ ls -l ~/.docker/buildx/instances
total 4
-rw-------. 1 jenkins-local-user jenkins-local-group 539 Aug 22 07:32 jenkins-multiarch

See Build variables | Docker Docs for reference!

Br.
Andreas

3

To further answer my own question about why it did not work in Jenkins the cause was that the docker jenkins plugins sets the DOCKER_CONFIG environment variable which docker buildx will use to try to find the builder instance but that will not work.

Example of working Jenkinsfile pipeline:

pipeline {
    agent any
    triggers {
        pollSCM ''
    }
    environment {
        BUILDX_BUILDER = 'jenkins-multiarch'
        BUILDX_CONFIG="${env.HOME}/.docker/buildx"
    }
    stages {
        stage('Build') {
            steps {
                withDockerRegistry([credentialsId: 'xxx', url: 'https://registry.xxx.net']) {
                    sh '@docker buildx build --push --platform linux/amd64,linux/arm64 --tag registry.xxx.net/image:latest'
                }
            }
        }
     }
}