Storage driver overlay over xfs -- ProjectQuota not reset on container delete

I’m trying to use overlay storage driver over xfs mount, to restrict the size of the container, But I’m facing the following issues

  • Whenever a container is created, there are 2 project quota id’s generated.
  • When container is removed, the quota is not removed and not reused for new containers.

This may lead to project id exhaustion on the xfs mount, has anyone else faced this issue or how to overcome this??

test@test:~$ sudo docker info
Client: Docker Engine - Community
 Version:    24.0.2
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc.)
    Version:  v0.10.5
    Path:     /usr/libexec/docker/cli-plugins/docker-buildx
  compose: Docker Compose (Docker Inc.)
    Version:  v2.18.1
    Path:     /usr/libexec/docker/cli-plugins/docker-compose

Server:
 Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
 Images: 1
 Server Version: 24.0.2
 Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Using metacopy: false
  Native Overlay Diff: true
  userxattr: false
 Logging Driver: journald
 Cgroup Driver: cgroupfs
 Cgroup Version: 1
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: io.containerd.runc.v2 runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 3dce8eb055cbb6872793272b4f20ed16117344f8
 runc version: v1.1.7-0-g860f061
 init version: de40ad0
 Security Options:
  apparmor
  seccomp
   Profile: builtin
 Kernel Version: 5.4.0-150-generic
 Operating System: Ubuntu 18.04.6 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 8
 Total Memory: 15.51GiB
 Name: zcode-3236
 ID: ee280665-0040-42b4-a382-24c4ff5b921b
 Docker Root Dir: /mnt/network_drive/code/xfs/Docker_Root
 Debug Mode: false
 Username: sarukazen
 Experimental: false
 Insecure Registries:
  localhost:8080
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: API is accessible on http://0.0.0.0:4243 without encryption.
         Access to the remote API is equivalent to root access on the host. Refer
         to the 'Docker daemon attack surface' section in the documentation for
         more information: https://docs.docker.com/go/attack-surface/
WARNING: No swap limit support
test@test:~$ sudo xfs_quota -x -c 'report -hbir' /mnt/xfs/
User quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
User ID      Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
root         856K      0      0  00 [------]     99      0      0  00 [------]      0      0      0  00 [------]
#501        19.6M      0      0  00 [------]    539      0      0  00 [------]      0      0      0  00 [------]
test           4K      0      0  00 [------]      4      0      0  00 [------]      0      0      0  00 [------]

Project quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
Project ID   Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
#0          20.4M      0      0  00 [------]    642      0      0  00 [------]      0      0      0  00 [------]

test@test:~$ sudo docker run -d --name xfstest --storage-opt size=1g ubuntu bash
cddf72659535dd364b1c1a124c7f42b44fb723f5f62b2bbb07be5ea715633ba8
test@test:~$ sudo xfs_quota -x -c 'report -hbir' /mnt/xfs/
User quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
User ID      Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
root        82.0M      0      0  00 [------]   3.7k      0      0  00 [------]      0      0      0  00 [------]
systemd-network      0      0      0  00 [------]      1      0      0  00 [------]      0      0      0  00 [------]
#501        19.6M      0      0  00 [------]    539      0      0  00 [------]      0      0      0  00 [------]
test           4K      0      0  00 [------]      4      0      0  00 [------]      0      0      0  00 [------]

Project quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
Project ID   Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
#0         101.6M      0      0  00 [------]   4.2k      0      0  00 [------]      0      0      0  00 [------]
#2             8K     1G     1G  00 [------]     17      0      0  00 [------]      0      0      0  00 [------]
#3             8K     1G     1G  00 [------]      6      0      0  00 [------]      0      0      0  00 [------]

test@test:~$ sudo docker rm xfstest
xfstest
test@test:~$ sudo xfs_quota -x -c 'report -hbir' /mnt/xfs/
User quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
User ID      Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
root        81.9M      0      0  00 [------]   3.6k      0      0  00 [------]      0      0      0  00 [------]
systemd-network      0      0      0  00 [------]      1      0      0  00 [------]      0      0      0  00 [------]
#501        19.6M      0      0  00 [------]    539      0      0  00 [------]      0      0      0  00 [------]
test           4K      0      0  00 [------]      4      0      0  00 [------]      0      0      0  00 [------]

Project quota on /mnt/xfs (/dev/nvme0n1p6)
                        Blocks                            Inodes                        Realtime Blocks          
Project ID   Used   Soft   Hard Warn/Grace     Used   Soft   Hard Warn/Grace    Used   Soft   Hard Warn/Grace   
---------- --------------------------------- --------------------------------- --------------------------------- 
#0         101.5M      0      0  00 [------]   4.2k      0      0  00 [------]      0      0      0  00 [------]
#2              0     1G     1G  00 [------]      0      0      0  00 [------]      0      0      0  00 [------]
#3              0     1G     1G  00 [------]      0      0      0  00 [------]      0      0      0  00 [------]

test@test:~$