Significantly refactor the gopls benchmarks to turn them into proper
benchmarks, eliminate the need for passing flags, and allow running them
on external gopls processes so that they may be used to test older gopls
versions.

Doing this required decoupling the benchmarks themselves from the
regtest.Runner. Instead, they just create their own regtest.Env to use
for scripting operations. In order to facilitate this, I tried to
redefine Env as a convenience wrapper around other primitives.

By using a separate environment setup for benchmarks, I was able to
eliminate a lot of regtest.Options that existed only to prevent the
regtest runner from adding instrumentation that would affect
benchmarking. This also helped clean up Runner.Run somewhat, though it
is still too complicated.

Also eliminate the unused AnyDiagnosticAtCurrentVersion, and make a few
other TODOs about future cleanup.

For golang/go#53992
For golang/go#53538

Change-Id: Idbf923178d4256900c3c05bc8999c0c9839a3c07
Reviewed-on: https://go-review.googlesource.com/c/tools/+/419988
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Peter Weinberger <pjw@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

For golang/go#53992

Change-Id: Ia1f1e27663992707eef9226273b152117ee977ac
Reviewed-on: https://go-review.googlesource.com/c/tools/+/420220
Run-TryBot: Robert Findley <rfindley@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Peter Weinberger <pjw@google.com>

Completion is very performance sensitive, and building a string to check
for *testing.F has a significant cost.

StructCompletion-8         20.7ms ±14%  16.8ms ± 1%  -18.59%  (p=0.000 n=10+10)
ImportCompletion-8         1.36ms ± 5%  1.05ms ±18%  -22.55%  (p=0.000 n=9+10)
SliceCompletion-8          23.5ms ± 2%  19.3ms ±18%  -17.85%  (p=0.000 n=7+10)
FuncDeepCompletion-8       17.6ms ± 2%  15.5ms ± 2%  -11.82%  (p=0.000 n=8+8)
CompletionFollowingEdit-8  81.2ms ± 8%  74.2ms ± 5%   -8.60%  (p=0.000 n=9+9)

For golang/go#53992
For golang/go#53798

Change-Id: Ia138cbadce142a424caabe8259bda05bcc536055
Reviewed-on: https://go-review.googlesource.com/c/tools/+/422906
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>

LookupFieldOrMethod appears as a hotspot when benchmarking gopls'
auto-completion. In particular, instanceLookup.add was allocating in the
common case of structs with no embedding.

This is easily fixed, by using a small array in front of the map inside
of instanceLookup. Do this, and additionally add a microbenchmark.

The benchmark improvement is significant:

name                    old time/op    new time/op    delta
LookupFieldOrMethod-12     388µs ± 6%     154µs ± 3%  -60.36%  (p=0.000 n=10+10)

name                    old alloc/op   new alloc/op   delta
LookupFieldOrMethod-12     152kB ± 0%       2kB ± 0%  -98.77%  (p=0.000 n=9+10)

name                    old allocs/op  new allocs/op  delta
LookupFieldOrMethod-12     1.41k ± 0%     0.07k ± 0%  -95.38%  (p=0.000 n=10+10)

It should also be noted that instanceLookup is used elsewhere, in
particular by validType. In those contexts, the scope is not just the
current type but the entire package, and so the newly added buffer is
likely to simply cause extra Identical checks. Nevertheless, those
checks are cheap, and on balance the improved LookupFieldOrMethod
performance leads overall to improved type-checking performance.
Standard library benchmark results varied by package, but type checking
speed for many packages improved by ~5%, with allocations improved by
~10%. If this weren't the case we could let the caller control the
buffer size, but that optimization doesn't seem necessary at this time.

For example:

Check/http/funcbodies/noinfo-12            71.5ms ± 4%    67.3ms ± 2%   -5.90%  (p=0.000 n=20+20)
Check/http/funcbodies/noinfo-12              244k ± 0%      219k ± 0%  -10.36%  (p=0.000 n=19+19)

Updates #53992

Change-Id: I10b6deb3819ab562dbbe1913f12b977cf956dd50
Reviewed-on: https://go-review.googlesource.com/c/go/+/423935
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@google.com>

LookupFieldOrMethod appears as a hotspot when benchmarking gopls'
auto-completion. In particular, instanceLookup.add was allocating in the
common case of structs with no embedding.

This is easily fixed, by using a small array in front of the map inside
of instanceLookup. Do this, and additionally add a microbenchmark.

The benchmark improvement is significant:

name                    old time/op    new time/op    delta
LookupFieldOrMethod-12     388µs ± 6%     154µs ± 3%  -60.36%  (p=0.000 n=10+10)

name                    old alloc/op   new alloc/op   delta
LookupFieldOrMethod-12     152kB ± 0%       2kB ± 0%  -98.77%  (p=0.000 n=9+10)

name                    old allocs/op  new allocs/op  delta
LookupFieldOrMethod-12     1.41k ± 0%     0.07k ± 0%  -95.38%  (p=0.000 n=10+10)

It should also be noted that instanceLookup is used elsewhere, in
particular by validType. In those contexts, the scope is not just the
current type but the entire package, and so the newly added buffer is
likely to simply cause extra Identical checks. Nevertheless, those
checks are cheap, and on balance the improved LookupFieldOrMethod
performance leads overall to improved type-checking performance.
Standard library benchmark results varied by package, but type checking
speed for many packages improved by ~5%, with allocations improved by
~10%. If this weren't the case we could let the caller control the
buffer size, but that optimization doesn't seem necessary at this time.

For example:

Check/http/funcbodies/noinfo-12            71.5ms ± 4%    67.3ms ± 2%   -5.90%  (p=0.000 n=20+20)
Check/http/funcbodies/noinfo-12              244k ± 0%      219k ± 0%  -10.36%  (p=0.000 n=19+19)

Updates golang#53992

Change-Id: I10b6deb3819ab562dbbe1913f12b977cf956dd50
Reviewed-on: https://go-review.googlesource.com/c/go/+/423935
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Robert Griesemer <gri@google.com>

Profiling revealed that scope lookup itself was costly, so memoize the
objects themselves, not just their names.

Also update BenchmarkCompletionFollowingEdit to allow it to be run on
multiple repos, and add a test case for kubernetes.

For golang/go#53992

Change-Id: I17b1f39d8c356e8628610a544306686573a813a7
Reviewed-on: https://go-review.googlesource.com/c/tools/+/502976
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
TryBot-Bypass: Robert Findley <rfindley@google.com>

No need to wait on xrefs or methodsets while performing
latency-sensitive operations on packages.

For golang/go#53992

Change-Id: I9ea65269a8c1e604fb99ed8b25e14db79f179576
Reviewed-on: https://go-review.googlesource.com/c/tools/+/503015
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>

The shared import graph exists to optimize type-checking of open
packages following a change. But we should only ever ask for ITVs when
type-checking the entire workspace (in the debounced call to
diagnoseSnapshot).

For golang/go#53992

Change-Id: Ie968d4b2a0476822999aa216c678bed370aaa67b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/503316
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>