Total CPU used by gopls is a critical metric for our users, yet was not
previously captured in any benchmark. This change uses the new pprof
parsing utilities added in CL 507885 to instrument a cpu_seconds
benchmark metric, for now just associated with the DidChange benchmark.

This is achieved via new LSP commands that start and stop a profile. The
benchmark runner uses these commands to bracket the critical section of
the benchmark, then parses the resulting profile for its total sampled
CPU.

Additionally, the benchmark runner is updated to actually check for the
existence of the custom command before instrumenting the new metric.
This allows it to be compatible with testing older versions of gopls.
The same technique is adopted for memstats metrics.

I only instrumented BenchmarkDidChange, because the profile file schema
is getting truly out of hand. I'll try to simplify it before
instrumenting all the other benchmarks, but want to do that in a
separate CL.

For golang/go#60926

Change-Id: Ia082bad49e8d30c567a7c07e050511d49b93738b
Reviewed-on: https://go-review.googlesource.com/c/tools/+/508449
gopls-CI: kokoro <noreply+kokoro@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

For the short term, we want to compare with v0.11.0. Hard code a
commit that includes additional instrumentation.

Add a TODO to select the latest patch release of the prior minor
version.

Updates golang/go#60926

Change-Id: If00a5a5f75d7a06af877dfdf2d78635a7ef3bd49
Reviewed-on: https://go-review.googlesource.com/c/build/+/509895
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

It is very confusing to have separate flags for instrumenting CPU
profiling of each test. Instead simplify to re-use the -gopls_cpuprofile
as a suffix that can be applied as necessary for each test. Some tests
want to instrument the entire gopls process, others want to use the new
profiling commands.

Also fix the "cpu_seconds" metric to be per-operation. It doesn't make
sense otherwise.

For golang/go#60926

Change-Id: I40e223e161abaa62b97b0021e480c7d6161e2853
Reviewed-on: https://go-review.googlesource.com/c/tools/+/509557
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>

Our existing benchmarks measure latency of various operations, but while
typing most operations do not run to completion: they are canceled. Add
a benchmark that simulates making changes at various fixed intervals,
and records the total CPU used.

For golang/go#60926

Change-Id: I972187358b6c802518ac781a12d9cff32faa6670
Reviewed-on: https://go-review.googlesource.com/c/tools/+/509559
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>

The getPackageHandles algorithm is performance critical, in that package
handles must be collected for any operation that involves type-checking.
Previously, it had suboptimal parallelism and involved several
unnecessary map lookups.

Rewrite it to avoid these unnecessary lookups, by operating directly on
package "index IDs", and improve parallism using a two-pass traversal
copied from the analysis driver.

Additionally, change the transitive ref computation to allow it to
proceed incrementally from invalid package handles. This optimizes the
case where a single high-level package is invalidated. We only need
the full set of transitive refs for direct dependencies; for indirect
dependencies, we may only need refs through a small subset of their
exported symbols.

Since my memory of this complicated logic had rotted a bit, add some
more documentation that would have helped.

For golang/go#60926

Change-Id: I7478acb1b36d0d0dc49d7631a0ca5713fd7e5373
Reviewed-on: https://go-review.googlesource.com/c/tools/+/509561
Auto-Submit: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Alan Donovan <adonovan@google.com>

The purpose of diagnosing changed files is to compute local diagnostics
with low latency. We only need one package for that. Don't slow
everything down by doing redundant and unnecessary work checking test
variants.

For golang/go#60926

Change-Id: I3d77305d5732fb0036e4865bbe7311d1561f2fd5
Reviewed-on: https://go-review.googlesource.com/c/tools/+/509876
Auto-Submit: Robert Findley <rfindley@google.com>
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

We were memoizing active packages only after a call to TypeCheck. This
means that we may duplicate work if diagnostics, references, or
implements requests execute before the first call to TypeCheck for an
open package.

Fix this by pushing the memoization logic down into forEachPackage.

This uncovered a bug in go/types that may have already been affecting
users: golang/go#61561. This bug is consistent with user reports on
slack of strange errors related to missing methods.

Avoid this bug in most cases by ensuring that all instantiated
interfaces are completed immediately after type checking. However, this
doesn't completely avoid the bug as it may not complete parameterized
interface literals elsewhere in the type definition. For example, the
following definition is still susceptible to the bug:

type T[P any] interface { m() interface{ n(P) } }

For golang/go#60926
Fixes golang/go#61005

Change-Id: I324cd13bac7c17b1eb5642973157bdbfb2368650
Reviewed-on: https://go-review.googlesource.com/c/tools/+/512636
Run-TryBot: Robert Findley <rfindley@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>