Adding GODEBUG to the test cache key seems appropriate — every Go program depends on GODEBUG via the runtime package, and it is likely that the runtime parses that variable in a way that is not visible to the test log.

Other environment variables go into the test-output cache key here:
https://cs.opensource.google/go/go/+/master:src/cmd/go/internal/test/test.go;l=1869-1870;drc=e076c1b897b30648db794b66fd10929fe9a9852e

GODEBUG is probably not being recorded their because it is parsed directly by the runtime package without going through the usual os.Getenv hooks. We should probably add it to the cache key explicitly.

A little confused here, -count=1 disables test caching explictly. (link)
This is the current behavior:

$go clean -cache

$GODEBUG=gotypesalias=1 go test ./doc

runtime
../../go/src/runtime/runtime2.go:993:6: internal compiler error: assertion failed
FAIL

$ go test ./doc

ok golang.org/x/tools/gopls/doc 4.267s

$ GODEBUG=gotypesalias=1 go test ./doc

ok golang.org/x/tools/gopls/doc (cached)

$ go test -count=1 ./doc

ok golang.org/x/tools/gopls/doc 4.281s

$ GODEBUG=gotypesalias=1 go test -count=1 ./doc

--- FAIL: TestGenerated (3.88s)
generate_test.go:26: documentation needs updating. Run: cd gopls && go generate
FAIL
FAIL golang.org/x/tools/gopls/doc 3.955s
FAIL

It seems to me something fishy is going on with the first line: GODEBUG=gotypesalias=1 go test ./doc fails but then succeeds using the cached test with GODEBUG=gotypesalias=1 omitted.

Do we want GODEBUG=gotypesalias=1 go test ./doc to not use the cached result from go test ./doc?

Do we want GODEBUG=gotypesalias=1 go test ./doc to not use the cached result from go test ./doc?

Yep, exactly. The mismatched GODEBUG setting should cause the cache key for the new test run not to match the prior run.

Change https://go.dev/cl/570259 mentions this issue: cmd/go: respect GODEBUG in test cache

Sorry if you were working on this @samthanawalla. I ran into this problem myself and had to fix it to make progress debugging something else. I didn't see this issue until after I'd written the CL.