Thanks to @renatorro, I was able to identify what causes that Disk leak.
For me it was because my Images were writing jar onto disk and every time I push a new version with like :
sudo kubectl set image deployment/imageName imageName=imageName:0-1-X
So the overlay2
folder increased each time.
I was able to clean it up by cordon my node, drain old unused image, and uncordon :
kubectl get nodes
Choose the your node in DiskPressure in the list, then detach it :
kubectl cordon
(It should appear as SchedulingDisabled
if you run kubectl get nodes
again)
Then drain all pods in the node :
kubectl drain --ignore-daemonsets
This could take a while if you have like me a lot of Evicted Pods…
You can now check the overlay2 folder size in the node :
cd /var/lib/docker
du -h --max-depth=1 | sort -hr
You should see the size diminishing while the drain is in progress…
When the drain is finished, you can re-attach your Node :
kubectl uncordon
The overlay folder should now re-increase to the normal size ( while your pods are loading)
Hope this can help someone !!