I found out that easiest way to clean up that directory (that in my case grew to 52GB in about 2 months) was to clean up the builder cache by issuing a:
docker builder prune
If you want to go one step further, use:
docker builder prune --all
Docs: https://docs.docker.com/engine/reference/commandline/builder_prune/