…r all of std

gcimporter.TestImportTypeparamTests still needs to create full export
data because it loads lots of source files from GOROOT/test that
expect to be able to import arbitrary subsets of the standard library,
so we now skip it in short mode.

On a clean build cache, this reduces
'go test -short cmd/compile/internal/importer go/internal/gcimporter'
on my machine from 21–28s per test to <6s per test.

Updates #56967.
Updates #47257.

Change-Id: I8fd80293ab135e0d2d213529b74e0ca6429cdfc7
Reviewed-on: https://go-review.googlesource.com/c/go/+/454498
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>

This fixes tests failing on the Android builders on Go 1.20 after CL
454499. Previously the tests were skipped in the 'compile' helper
function, but as of that CL they fail before reaching that point due
to missing export data for packages in std.

Updates golang/go#56967.
Updates golang/go#47257.

Change-Id: Ief953b6dbc54c8e0b1f71fc18a0d6ab212caf308
Reviewed-on: https://go-review.googlesource.com/c/tools/+/454500
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>

…dLib

The test attempted to find all stdlib packages by scanning
pkg/$GOOS_$GOARCH for .a files and then tried to import all of them.
Now that .a files are no longer being placed there, the test is a
noop. Fix this by using go list std (and filtering out testonly
packages) and trying to import all of those to recreate what the test
intended to do.

This also removes a dependency on the pkg/$GOOS_$GOARCH directory
which will stop being produced by dist in CL 453496.

Updates golang/go#47257

Change-Id: Idfa0cbb21093776183ce193eb5363a9727bf77ef
Reviewed-on: https://go-review.googlesource.com/c/tools/+/454118
Run-TryBot: Michael Matloob <matloob@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Bryan Mills <bcmills@google.com>
(cherry picked from commit 0379b73)
Reviewed-on: https://go-review.googlesource.com/c/tools/+/462636
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Russ Cox <rsc@golang.org>

…rt data for packages individually

In short tests, also avoid creating export data for all of std.

This change applies the same improvements made to the equivalent std
packages in CL 454497 and CL 454498. (It may even fix the reverse
builders that are currently timing out in x/tools, although I suspect
there is still a bit of work to do for those.)

Updates golang/go#56967.
Updates golang/go#47257.

Change-Id: I82e72557da5f917203637513122932c7942a98e9
Reviewed-on: https://go-review.googlesource.com/c/tools/+/454499
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Matloob <matloob@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
(cherry picked from commit f540ee6)
Reviewed-on: https://go-review.googlesource.com/c/tools/+/462637
Reviewed-by: Russ Cox <rsc@golang.org>

…er when 'go build' is not available

This fixes tests failing on the Android builders on Go 1.20 after CL
454499. Previously the tests were skipped in the 'compile' helper
function, but as of that CL they fail before reaching that point due
to missing export data for packages in std.

Updates golang/go#56967.
Updates golang/go#47257.

Change-Id: Ief953b6dbc54c8e0b1f71fc18a0d6ab212caf308
Reviewed-on: https://go-review.googlesource.com/c/tools/+/454500
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Jamal Carvalho <jamal@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
(cherry picked from commit bdcd082)
Reviewed-on: https://go-review.googlesource.com/c/tools/+/462638
Reviewed-by: Russ Cox <rsc@golang.org>

Since packages in "std" no longer have install targets, checking them
for staleness is somewhat meaningless: if they are not cached they
will be rebuilt anyway, and without installed archives against which
we can compare them the staleness check will not detect builder skew.

It would still be meaningful to check "cmd" for staleness, but
(especially on sharded VM-based builders) that is a fairly expensive
operation relative to its benefit. If we are really interested in
detecting builder skew and/or build reproducibility, we could instead
add a "misc" test (similar to "misc/reboot", or perhaps even a part of
that test) that verifies that bootstrapped binaries are reproducible.

For #57734.
Updates #47257.
Updates #56896.

Change-Id: I8683ee81aefe8fb59cce9484453df9729bdc587c
Reviewed-on: https://go-review.googlesource.com/c/go/+/452775
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Austin Clements <austin@google.com>

The purpose of building the host toolchain is so that we can use it to
build and test the target configuration.

The host configuration should already be tested separately (with its
own builder), so we do not need to build the parts of that
configuration that are not relevant to the task of building and
testing the target configuration.

Updates #47257.

Change-Id: I814778d2d65b1f2887c9419232b5bfd4068f58af
Reviewed-on: https://go-review.googlesource.com/c/go/+/461676
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>

Instead, have the caller pass in an explicit list of the packages
(if any) they need.

After #47257, a builder running a test does not necessarily have the
entire standard library already cached, especially when running tests
in sharded mode. testenv.WriteImportcfg used to write an importcfg for
the entire standard library — which required rebuilding the entire
standard library — even though most tests need only a tiny subset.

This reduces the time to test internal/abi with a cold build cache on
my workstation from ~16s to ~0.05s.

It somewhat increases the time for 'go test go/internal/gcimporter'
with a cold cache, from ~43s to ~54s, presumably due to decreased
parallelism in rebuilding the standard library and increased overhead
in re-resolving the import map. However, 'go test -short' running time
remains stable (~5.5s before and after).

Fixes #58248.

Change-Id: I9be6b61ae6e28b75b53af85207c281bb93b9346f
Reviewed-on: https://go-review.googlesource.com/c/go/+/464736
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>

Since packages in "std" no longer have install targets, checking them
for staleness is somewhat meaningless: if they are not cached they
will be rebuilt anyway, and without installed archives against which
we can compare them the staleness check will not detect builder skew.

It would still be meaningful to check "cmd" for staleness, but
(especially on sharded VM-based builders) that is a fairly expensive
operation relative to its benefit. If we are really interested in
detecting builder skew and/or build reproducibility, we could instead
add a "misc" test (similar to "misc/reboot", or perhaps even a part of
that test) that verifies that bootstrapped binaries are reproducible.

For golang#57734.
Updates golang#47257.
Updates golang#56896.

Change-Id: I8683ee81aefe8fb59cce9484453df9729bdc587c
Reviewed-on: https://go-review.googlesource.com/c/go/+/452775
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Austin Clements <austin@google.com>

The purpose of building the host toolchain is so that we can use it to
build and test the target configuration.

The host configuration should already be tested separately (with its
own builder), so we do not need to build the parts of that
configuration that are not relevant to the task of building and
testing the target configuration.

Updates golang#47257.

Change-Id: I814778d2d65b1f2887c9419232b5bfd4068f58af
Reviewed-on: https://go-review.googlesource.com/c/go/+/461676
Run-TryBot: Bryan Mills <bcmills@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Austin Clements <austin@google.com>

Instead, have the caller pass in an explicit list of the packages
(if any) they need.

After golang#47257, a builder running a test does not necessarily have the
entire standard library already cached, especially when running tests
in sharded mode. testenv.WriteImportcfg used to write an importcfg for
the entire standard library — which required rebuilding the entire
standard library — even though most tests need only a tiny subset.

This reduces the time to test internal/abi with a cold build cache on
my workstation from ~16s to ~0.05s.

It somewhat increases the time for 'go test go/internal/gcimporter'
with a cold cache, from ~43s to ~54s, presumably due to decreased
parallelism in rebuilding the standard library and increased overhead
in re-resolving the import map. However, 'go test -short' running time
remains stable (~5.5s before and after).

Fixes golang#58248.

Change-Id: I9be6b61ae6e28b75b53af85207c281bb93b9346f
Reviewed-on: https://go-review.googlesource.com/c/go/+/464736
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Cherry Mui <cherryyz@google.com>
Reviewed-by: Than McIntosh <thanm@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>

…l compile' does not exist

Now that .a files are no longer being preinstalled for
standard-library packages (#47257), tests that require the export data
from those files must have access to 'go list -export' in order to
generate and load export data from the Go build cache.

`go list -export` does not work without a Go compiler, so assume that
tests that need the 'go' command also need the compiler.

This may cause the tests currently failing on the android-.*-emu
builders to instead be skipped, since the test harness does not copy
the compiler to the execution environment.

For golang/go#47257.

Change-Id: Ie82ab09ac8165a01fc1d3a083b1e86226efc469d
Reviewed-on: https://go-review.googlesource.com/c/tools/+/451597
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Matloob <matloob@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>

1.19.10 includes a change on linked libraries that's messing up our manager binary

see golang/go#47257 and golang/go#57328
(issues are closed, but the error is the same)