Docker rootless mode / traefik permission pb

Hello,
I am completely lost and fighting with this pb for weeks.
I am using Traefik and crowdsec using compose file until I manage to use them on rootless mode.
So new proxmox VM, installation of the docker in rootless mode, it seems to be so fare ok.

~$ docker context show
rootless
theboss@proxmox-ve-node1-serverdockers:~$

Few weeks ago, i have installed Gotify to send me notifications if there is any input from Crowdsec … I was a bit surprised not to have any notification. I checked on the security engine and see that there were 0 alert …
Then I checked the logs’s traefik and saw that it was not normal … Nothing on it when IO was testing with my web site.
My traefik logs today:

2024-05-15T15:59:29+03:00 ERR Plugins are disabled because an error has occurred. error="unable to create plugins client: unable to create directory /plugins-storage/sources: mkdir plugins-storage: read-only file system"
2024-05-15T15:59:29+03:00 INF Starting provider aggregator aggregator.ProviderAggregator
2024-05-15T15:59:29+03:00 INF Starting provider *file.Provider
2024-05-15T15:59:29+03:00 INF Starting provider *traefik.Provider
2024-05-15T15:59:29+03:00 INF Starting provider *acme.ChallengeTLSALPN
2024-05-15T15:59:29+03:00 INF Starting provider *docker.Provider
2024-05-15T15:59:29+03:00 INF Starting provider *acme.Provider
2024-05-15T15:59:29+03:00 INF Testing certificate renew... acmeCA=https://acme-v02.api.letsencrypt.org/directory providerName=dns-cloudflare.acme

The volumes of my traefik compose file

services:
  traefik:
    image: traefik:v3.0.0
    container_name: traefik
    restart: unless-stopped
    security_opt:
      - no-new-privileges:true
    read_only: true
    mem_limit: 2G
    cpus: 0.75
    depends_on:
      - dockerproxy
    networks:
      - $MYNET
      - socket-t
  
    ports:
      - 1180:80
      - 11443:443
      - 8087:8080
      - 1181:1181
      - 11444:11444
    environment:
      CF_API_EMAIL: $EMAIL
      CF_DNS_API_TOKEN: $TOKEN
      TZ: Europe/Helsinki
      GID: ${GID-988}
      
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - $BASE/logs/crowdsec/:$BASE/logs/crowdsec/
      - $BASE/traefik/data/traefik.yml:/traefik.yml:ro
      - $BASE/traefik/data/dynamic_conf.yml:/dynamic_conf.yml:ro
      - $BASE/letsencrypt:/letsencrypt


    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=$MYNET"
      - "traefik.http.routers.traefik.entrypoints=http"
      - "traefik.http.routers.traefik.rule=Host(`traefik.$MYDOMAIN`)"
      - "traefik.http.middlewares.traefik-https-redirect.redirectscheme.scheme=https"
      - "traefik.http.middlewares.sslheader.headers.customrequestheaders.X-Forwarded-Proto=https"
      - "traefik.http.routers.traefik.middlewares=traefik-https-redirect"
      - "traefik.http.routers.traefik-secure.entrypoints=https"
      - "traefik.http.routers.traefik-secure.rule=Host(`traefik.$MYDOMAIN`)"
      - "traefik.http.routers.traefik-secure.middlewares=traefik-auth"
      - "traefik.http.routers.traefik-secure.tls=true"
      - "traefik.http.routers.traefik-secure.tls.certresolver=dns-cloudflare"
      - "traefik.http.routers.traefik-secure.tls.domains[0].main=$MYDOMAIN"
      - "traefik.http.routers.traefik-secure.tls.domains[0].sans=*.$MYDOMAIN"
      - "traefik.http.routers.traefik-secure.service=api@internal"
      # middlewares
      - "traefik.http.middlewares.traefik-auth.basicauth.removeheader=true"
      - "traefik.http.middlewares.traefik-auth.basicauth.users=login:xxxxxxxxxxxx"
      # middlewares security headers
      - "traefik.http.middlewares.security-headers.headers.accesscontrolallowmethods=GET, OPTIONS, PUT"
      - "traefik.http.middlewares.security-headers.headers.accesscontrolmaxage=100"
      - "traefik.http.middlewares.security-headers.headers.addvaryheader=true"
      - "traefik.http.middlewares.security-headers.headers.hostsproxyheaders=X-Forwarded-Host"
      - "traefik.http.middlewares.security-headers.headers.sslredirect=true"
      - "traefik.http.middlewares.security-headers.headers.sslproxyheaders.X-Forwarded-Proto=https"
      - "traefik.http.middlewares.security-headers.headers.stsseconds=63072000"
      - "traefik.http.middlewares.security-headers.headers.stsincludesubdomains=true"
      - "traefik.http.middlewares.security-headers.headers.stspreload=true"
      - "traefik.http.middlewares.security-headers.headers.forcestsheader=true"
      - "traefik.http.middlewares.security-headers.headers.framedeny=true"
      - "traefik.http.middlewares.security-headers.headers.contenttypenosniff=true"
      - "traefik.http.middlewares.security-headers.headers.browserxssfilter=true"
      - "traefik.http.middlewares.security-headers.headers.referrerpolicy=same-origin"
      - "traefik.http.middlewares.security-headers.headers.featurepolicy=camera 'none'; geolocation 'none'; microphone 'none'; payment 'none'; usb 'none'; vr 'none';"
      - "traefik.http.middlewares.security-headers.headers.customresponseheaders.X-Robots-Tag=none,noarchive,nosnippet,notranslate,noimageindex"


  dockerproxy:
    image: wollomatic/socket-proxy:1.3.1
    container_name: t-docker-socket-proxy
    command:
      - '-loglevel=debug'
      - '-allowfrom=0.0.0.0/0'
      - '-listenip=0.0.0.0'
      - '-allowGET=/v1\..{1,2}/(version|containers/.*|events.*)'
      - '-watchdoginterval=3600'
      - '-stoponwatchdog'
      - '-shutdowngracetime=10'
    restart: unless-stopped
    read_only: true
    mem_limit: 64M
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges
    user: 65534:988 # change gid from 998 to the gid of the docker group on your host
    volumes:
      #- /var/run/docker.sock:/var/run/docker.sock:ro
      - /run/user/1000/docker.sock:/var/run/docker.sock:ro
    networks:
      - socket-t

networks:
  mynet:
    name: $MYNET
    external: true
  socket-t:
    driver: bridge
    internal: true
    attachable: false

Thx

Did you read the error message?

Traefik wants to create a folder, but the image is read-only and the path is not a writable bind mount.

Hello @bluepuma77 are you on both forum ? :slight_smile:
Yes I have of course read this error message … I know there is a permissions pb with a mounted volume.
I have to resolve first the problem concerning the folder “plugins-storage”, what PATH should it use ??
So is it a docker problem because running in rootless mode ?? Or is it a pb of config with Traefik because of this rootless mode ??

I will still be on it today.
Thx

I might have found something …

user@pproxserverdockers:~/.local/share/docker$ ll
total 28
drwx--x---  13 user user   189 mai   12 02:57 ./
drwx--x--x   3 user user    20 mai   10 08:28 ../
drwx--x--x   4 user user   138 mai   10 08:28 buildkit/
drwx--x--x   3 user user    20 mai   10 08:28 containerd/
drwx--x---  21 user user  4096 mai   16 05:07 containers/
-rw-------   1 user user    36 mai   10 08:28 engine-id
drwx------   3 user user    22 mai   10 08:28 image/
drwxr-x---   3 user user    19 mai   10 08:28 network/
drwx--x--- 170 user user 16384 mai   16 05:07 overlay2/
drwx------   4 user user    32 mai   10 08:28 plugins/
drwx------   2 user user     6 mai   12 02:57 runtimes/
drwx------   2 user user     6 mai   10 08:28 swarm/
drwx------   3 user user    39 mai   14 05:33 tmp/
drwx-----x  15  100999  100987   245 mai   16 05:05 volumes/
user@pproxserverdockers:~/.local/share/docker$ id
uid=1000(user) gid=1000(user) groups=1000(user),4(adm),24(cdrom),27(sudo),30(dip),46(plugdev),101(lxd)
user@pproxserverdockers:~/.local/share/docker$

With the user “1000” I have access up to “docker folder” … After I do have a permission deny.
Whe you see the user and group for “volumes” it is 100999 / 100987 … Where is this coming from ?

From “volumes”

user@pproxserverdockers:~/.local/share/docker/volumes$ ll
ls: cannot open directory '.': Permission denied
user@pproxserverdockers:~/.local/share/docker/volumes$

After “volumes” when in “root”

root@pproxserverdockers:/home/user/.local/share/docker/volumes# ll
total 28
drwx-----x 15  100999  100987   245 mai   16 05:05 ./
drwx--x--- 13 user user   189 mai   12 02:57 ../
drwxr-xr-x  8  100999  100987    98 mai   12 03:08 authentik/
drwx-----x  3  100999  100987    19 mai   10 16:13 authentik_redis/
drwxr-xr-x  3  100999  100987    20 mai   10 12:22 code-server/
drwxr-xr-x 14  100999  100987  4096 mai   14 02:29 crowdsec/
drwxr-xr-x  3  100999  100987    18 mai   10 15:49 diun/
drwxr-xr-x  4  100999  100987    52 mai   16 05:09 gotify/
drwxr-xr-x  2  100999  100987    23 mai   11 09:21 letsencrypt/
drwxr-xr-x  3  100999  100987    22 mai   10 09:02 logs/
-rw-------  1  100999  100987 32768 mai   13 14:09 metadata.db
drwx-----x  3  100999  100987    19 mai   10 08:53 portainer_data/
drwxr-xr-x  3  100999  100987    18 mai   10 15:30 rustdesk/
drwxr-xr-x  4  100999  100987    41 mai   15 05:20 traefik/
drwxr-xr-x  3  100999  100987    18 mai   14 05:58 uptime-kuma/
root@pproxserverdockers:/home/user/.local/share/docker/volumes#```

ideas ??

Only quickly tried rootless, experienced too much pain with access rights.

You would need to create the users on each node, assign the proper rights to folders and then pass the user to the container.

Or check which special user id a container defaults to and create according user on the host and assign proper rights to folders.

Not sure where your user ids are coming from.

https://docs.docker.com/engine/security/rootless/#prerequisites

Rootless container means Docker will use a user namespace. Your user is UID 0 inside the containers so you don’t need root to be able to create something as root in the container and still work outside the container. The other IDs are the IDs in the user namespace.

The linked documentation explains how you can get the smallest UID in the user namespace and the number of IDs that can be used in that namespace. IDs you can see on the host are smallest ID + ID in the container. If the smallest is 100000, then it is UID 1 inside the container and UID 100001 outside is UID 2 inside the container. You have to keep it in mind when you bind mount a folder from the host. And also that when you want to mount a folder where your user is not the owner and can’t write it, the container will not be able to write it either, unless the owner is in the user namespace.

You can use rootlesskit to execute a command in the user namespace. It is alos mentioned in the documentation. You can even start a shell:

rootlesskit sh
1 Like