Go version

go version go1.21.4 linux/amd64

Output of go env in your module/workspace:

GOCACHE=/foo/deps/gomod
GOMODCACHE=/foo/deps/gomod/pkg/mod
GOPATH=/foo/deps/gomod
GOPROXY=file:///foo/deps/gomod/pkg/mod/cache/download
GOTOOLCHAIN=auto

What did you do?

1. Pre-fetch dependencies of a project and store them in the mod cache favoring the latest toolchain on a given arch (x86) using GOTOOLCHAIN=auto:
2. Run a build on a different arch (aarch64) using the mod cache from previous step with no network:

What did you see happen?

1. on an online x86_64 machine:

$ go version
go version go1.21.4 linux/amd64

$ cat go.mod
go 1.21
toolchain 1.21.6
...

$ GOTOOLCHAIN=auto GOSUMDB=sum.golang.org ... go mod download
go: downloading go1.21.6 (linux/amd64)
...

2. on an offline aarch64 machine:

$ go version
go version go1.21.4 linux/arm64

$ GOTOOLCHAIN=auto GOMODCACHE=/foo/deps/gomod/pkg/mod GOPROXY=file:///foo/deps/gomod/pkg/mod/cache/download ... go build -a -x -v
go: downloading go1.21.6 (linux/arm64)
go: download go1.21.6 for linux/arm64: toolchain not available

$ tree /foo/deps/gomod| grep -C5 toolchain
...
cache
  └ download
        ├ golang.org
              ├ toolchain
                    └ @v
                           ├ v0.0.1-go1.21.6.linux-amd64.lock
                           ├ **v0.0.1-go1.21.6.linux-amd64.zip**
                           ├ v0.0.1-go1.21.6.linux-amd64.ziphash
                           └ **v0.0.1-go1.21.6.linux-arm64.lock**
...

What did you expect to see?

Given that one is able to fetch and download a given release of Go for a given arch manually successfully:
GOARCH=arm64 go install 'golang.org/dl/go1.21.8@latest'; <path>/go/bin/linux_arm64/go1.21.8

I would assume that by instrumenting Go with GOARCH=<arch> GOTOOLCHAIN=auto with e.g. go mod download Go would also fetch a toolchain of the desired version for the desired architecture, but instead a native toolchain of the desired version is always downloaded instead, naturally failing the build on the other arch. Now, aside from Go being able to cross-compile for many platform/arch pairs out-of-the box already I'm wondering if you could consider this use case as well (have a single node to prefetch artifacts for the rest of the nodes in a multi-arch node setup) or there's some major technical blocker rendering this as a complete non-issue not worth addressing.