Problems building multi-arch images with Bake and multiple buildx drivers

I am attempting to build an image use docker buildx bake for multiple architectures. Here is the bake file:

target "base-linux-amd64" {
    context = "."
    target = "base-amd"
    platforms = ["linux/amd64"]
    tags = ["localhost:5000/base-amd:latest"]
}

target "base-linux-arm64" {
    context = "."
    target = "base-arm"
    platforms = ["linux/arm64"]
    tags = ["localhost:5000/base-arm:latest"]
}

target "cross-ubuntu" {
    context = "."
    platforms = ["linux/amd64", "linux/arm64"]
    contexts = {
        "base-image": "target:base-${replace(BAKE_LOCAL_PLATFORM, "/", "-")}"
    }
    tags = [
        "localhost:5000/bake-test:latest"
    ]
}

Here is the buildx driver I am using:

composite*            docker-container                                        
 \_ amd64              \_ unix:///var/run/docker.sock    running   v0.15.1    linux/amd64*, linux/amd64/v2, linux/amd64/v3, linux/arm64, linux/riscv64, linux/ppc64, linux/ppc64le, linux/s390x, linux/386, linux/mips64le, linux/mips64, linux/arm/v7, linux/arm/v6
 \_ arm64              \_ ssh://t1user@block-np          running   v0.15.1    linux/arm64*, linux/arm/v7, linux/arm/v6

As can be seen, this driver has two nodes, one is a docker-container driver on the current machine (which is linux/amd64) and the other is a driver on a remote ARM machine added via ssh.

Given this setup, I can run docker buildx bake -f docker-bake.hcl base-linux-amd64 --push and docker buildx bake -f docker-bake.hcl base-linux-arm64 --push without issue. But when I run docker buildx bake -f docker-bake.hcl cross-ubuntu --push, I get:

ERROR: failed to use base-linux-amd64 as context base-image for cross-ubuntu because targets build with different drivers

What is the rational behind forbidding dependencies across drivers like this? Isn’t the context being depended on a docker image, so why does it matter that it came from a separate driver? And what is the best way to work around this while still using bake (for concurrency) and separate driver nodes?