Windows Container Dockerfile "Removing Intermediate Containers" Steps Takes Long Time

When I install a lot of node_modules in my windows container, it takes a very long time to remove the intermediate container in the build process. Has anyone else experienced this?
I’m using Docker For Windows 2.4.0.0 stable

I don’t think the issue lies in the specific image though, I’ve used the image below and have the same issues:

Did you ever find a solution to this problem?