When supplying many platforms at once to docker buildx build…, then the threads can consume a lot of memory. This doesn’t work well for medium (1 GB RAM per image build) application image builds. On my macOS host, attempting to compile clang in Linux images across all the available Docker platforms, completely stalls overnight, requiring more than 15 GB of RAM on my 16 GB host. Whereas an individual docker build thread with no buildx concurrency or multi-platform images, is able to successfully complete with limited resources.
For comparison, note that many programs that offer concurrency, like make, provide a command line flag to batch the concurrent tasks in smaller concurrent groups. And they provide a sane default, such as 4 tasks at a time, or the number of CPU cores at a time. When can we get a similar option for the user to control buildx build max threads? A command line flag or environment variable here would be really helpful for local build operations.
I tried prefixing my docker buildx build commands with GOMAXPROCS=4 , but that appears to have no effect. Likely the buildx tasks run as goroutines. But there should really be some way to restrict the number of concurrent buildx tasks.
As a workaround, I am manually breaking up the platforms into separate batch commands. This can be accomplished in a crude form with a development build of the tug wrapper:
However, any previously cached builds make it difficult for an external program to batch intelligently: There will be some waste. Rather, batching buildx tasks is best implemented by Docker itself, with a -j flag or environment variable.