As shown in the screenshot, Docker for Mac requires at least 1GB memory, which is undesirable for my use case (mainly docker pull & docker push to transfer images + occasionally running small CLIs).
Wow, this forum is really helpful. I somehow figured it out myself - the settings are stored at ~/Library/Group Containers/group.com.docker/settings.json, just set memoryMiB to whatever you like.