As you already figured out, the behavior is caused by how WSL2 handles the vhdx file. It is a dynamic growing virtual hard disk file with max size of 256gb by default.
In the pasts I did shrink vhdx files by following a solution like this: How to Shrink a WSL2 Virtual Disk – Stephen Rees-Carter
It could be actually a usefully feature-request to have a button in Docker Desktop that shrinks the used vhdx file.