This lifts the DAG parser from the go/build dependencies test into its
own package that can be reused elsewhere.

I tried to keep the code as close as possible. I changed some names to
reflect the more general purpose of internal/dag. Most of the changes
are related to error handling, since internal/dag doesn't take a
testing.T on which to report errors. Notably, parseRules now returns a
slice of parsed rules rather than calling a callback because this made
it easier to separate fatal parsing errors from non-fatal graph
checking errors.

For #53789.

Change-Id: I170b84fd85f971cfc1a50972156d48e78b45fce3
Reviewed-on: https://go-review.googlesource.com/c/go/+/418592
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

The go/types package doesn't care about node ordering because it's
just querying paths in the graph, but we're about to use this for the
runtime lock graph, and there we want determinism.

For #53789.

Change-Id: Ic41329bf2eb9a3a202f97c21c761ea588ca551c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/418593
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>

For #53789.

Change-Id: Ic7379afcfdcc47b541bac9b44b5bc6b43604fc0a
Reviewed-on: https://go-review.googlesource.com/c/go/+/418714
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>

We're missing lock edges to trace.lock that happen only rarely. Any
trace event can potentially fill up a trace buffer and acquire
trace.lock in order to flush the buffer, but this happens relatively
rarely, so we simply haven't seen some of these lock edges that could
happen.

With this change, we promote "fin, notifyList < traceStackTab" to
"fin, notifyList < trace" and now everything that emits trace events
with a P enters the tracer lock ranks via "trace", rather than some
things entering at "trace" and others at "traceStackTab".

This was found by inspecting the rank graph for things that didn't
make sense.

Ideally we would add a mayAcquire annotation that any trace event can
potentially acquire trace.lock, but there are actually cases that
violate this ranking right now. This is #53979. The chance of a lock
cycle is extremely low given the number of conditions that have to
happen simultaneously.

For #53789.

Change-Id: Ic65947d27dee88d2daf639b21b2c9d37552f0ac0
Reviewed-on: https://go-review.googlesource.com/c/go/+/418716
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

We're missing lock edges to finlock that happen only rarely. Anything
that calls mallocgc can potentially trigger sweeping, which can
potentially queue a finalizer, which acquires finlock. While this can
happen on any malloc, it happens relatively rarely, so we simply
haven't seen some of the lock edges that could happen.

Add a mayAcquire annotation to mallocgc to capture the possibility of
acquiring finlock.

With this change, we add "fin" to the set of "malloc" locks. Several
of these edges were already there, but not quite all of them.

This was found by inspecting the rank graph for things that didn't
make sense.

For #53789.

Change-Id: Idc10ce6f250596b0c07ba07ac93f2198fb38c22b
Reviewed-on: https://go-review.googlesource.com/c/go/+/418717
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

This groups, comments, and generally reorganizes the lock rank graph
description by subsystem. It also introduces several pseudo-nodes that
more cleanly describe the inherent layering of lock ranks by
subsystem.

I believe this doesn't actually change the graph, but haven't verified
this.

For #53789.

Change-Id: I72f332f5a23b8217c7dc1b21411631ad48cee4b0
Reviewed-on: https://go-review.googlesource.com/c/go/+/418718
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>

I'm not entirely sure why these locks are currently ranked "deadlock <
panic" since we drop panic before acquiring deadlock, and we actually
want deadlock to be below panic because panic is implicitly below
everything else and we want deadlock to be, too. My best guess is that
we had this edge because we intentionally acquire deadlock twice to
deadlock, and that causes the lock rank checking to panic on the
second acquire.

Fix this in a more sensible way by capturing that deadlock can be
acquired in a self-cycle and flipping the rank to "panic < deadlock"
to express that deadlock needs to be under all other locks, just like
panic.

For #53789.

Change-Id: I8809e5d102ce473bd3ace0ba07bf2200ef60263f
Reviewed-on: https://go-review.googlesource.com/c/go/+/418719
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>

This lifts the DAG parser from the go/build dependencies test into its
own package that can be reused elsewhere.

I tried to keep the code as close as possible. I changed some names to
reflect the more general purpose of internal/dag. Most of the changes
are related to error handling, since internal/dag doesn't take a
testing.T on which to report errors. Notably, parseRules now returns a
slice of parsed rules rather than calling a callback because this made
it easier to separate fatal parsing errors from non-fatal graph
checking errors.

For golang#53789.

Change-Id: I170b84fd85f971cfc1a50972156d48e78b45fce3
Reviewed-on: https://go-review.googlesource.com/c/go/+/418592
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

The go/types package doesn't care about node ordering because it's
just querying paths in the graph, but we're about to use this for the
runtime lock graph, and there we want determinism.

For golang#53789.

Change-Id: Ic41329bf2eb9a3a202f97c21c761ea588ca551c8
Reviewed-on: https://go-review.googlesource.com/c/go/+/418593
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>

For golang#53789.

Change-Id: Ic7379afcfdcc47b541bac9b44b5bc6b43604fc0a
Reviewed-on: https://go-review.googlesource.com/c/go/+/418714
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>

Currently, the runtime lock rank graph is maintained manually in a
large set of arrays that give the partial order and a manual
topological sort of this partial order. Any changes to the rank graph
are difficult to reason about and hard to review, as well as likely to
cause merge conflicts. Furthermore, because the partial order is
manually maintained, it's not actually transitively closed (though
it's close), meaning there are many cases where rank a can be acquired
before b and b before c, but a cannot be acquired before c. While this
isn't technically wrong, it's very strange in the context of lock
ordering.

Replace all of this with a much more compact, readable, and
maintainable description of the rank graph written in the internal/dag
graph language. We statically generate the runtime structures from
this description, which has the advantage that the parser doesn't have
to run during runtime initialization and the structures can live in
static data where they can be accessed from any point during runtime
init.

The current description was automatically generated from the existing
partial order, combined with a transitive reduction. This ensures it's
correct, but it could use some manual messaging to call out the
logical layers and add some structure.

We do lose the ad hoc string names of the lock ranks in this
translation, which could mostly be derived from the rank constant
names, but not always. I may bring those back but in a more uniform
way.

We no longer need the tests in lockrank_test.go because they were
checking that we manually maintained the structures correctly.

Fixes golang#53789.

Change-Id: I54451d561b22e61150aff7e9b8602ba9737e1b9b
Reviewed-on: https://go-review.googlesource.com/c/go/+/418715
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

We're missing lock edges to trace.lock that happen only rarely. Any
trace event can potentially fill up a trace buffer and acquire
trace.lock in order to flush the buffer, but this happens relatively
rarely, so we simply haven't seen some of these lock edges that could
happen.

With this change, we promote "fin, notifyList < traceStackTab" to
"fin, notifyList < trace" and now everything that emits trace events
with a P enters the tracer lock ranks via "trace", rather than some
things entering at "trace" and others at "traceStackTab".

This was found by inspecting the rank graph for things that didn't
make sense.

Ideally we would add a mayAcquire annotation that any trace event can
potentially acquire trace.lock, but there are actually cases that
violate this ranking right now. This is golang#53979. The chance of a lock
cycle is extremely low given the number of conditions that have to
happen simultaneously.

For golang#53789.

Change-Id: Ic65947d27dee88d2daf639b21b2c9d37552f0ac0
Reviewed-on: https://go-review.googlesource.com/c/go/+/418716
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

We're missing lock edges to finlock that happen only rarely. Anything
that calls mallocgc can potentially trigger sweeping, which can
potentially queue a finalizer, which acquires finlock. While this can
happen on any malloc, it happens relatively rarely, so we simply
haven't seen some of the lock edges that could happen.

Add a mayAcquire annotation to mallocgc to capture the possibility of
acquiring finlock.

With this change, we add "fin" to the set of "malloc" locks. Several
of these edges were already there, but not quite all of them.

This was found by inspecting the rank graph for things that didn't
make sense.

For golang#53789.

Change-Id: Idc10ce6f250596b0c07ba07ac93f2198fb38c22b
Reviewed-on: https://go-review.googlesource.com/c/go/+/418717
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>

This groups, comments, and generally reorganizes the lock rank graph
description by subsystem. It also introduces several pseudo-nodes that
more cleanly describe the inherent layering of lock ranks by
subsystem.

I believe this doesn't actually change the graph, but haven't verified
this.

For golang#53789.

Change-Id: I72f332f5a23b8217c7dc1b21411631ad48cee4b0
Reviewed-on: https://go-review.googlesource.com/c/go/+/418718
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Austin Clements <austin@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>

I'm not entirely sure why these locks are currently ranked "deadlock <
panic" since we drop panic before acquiring deadlock, and we actually
want deadlock to be below panic because panic is implicitly below
everything else and we want deadlock to be, too. My best guess is that
we had this edge because we intentionally acquire deadlock twice to
deadlock, and that causes the lock rank checking to panic on the
second acquire.

Fix this in a more sensible way by capturing that deadlock can be
acquired in a self-cycle and flipping the rank to "panic < deadlock"
to express that deadlock needs to be under all other locks, just like
panic.

For golang#53789.

Change-Id: I8809e5d102ce473bd3ace0ba07bf2200ef60263f
Reviewed-on: https://go-review.googlesource.com/c/go/+/418719
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Austin Clements <austin@google.com>

Now that we've moved the trace locks to the leaf of the lock graph, we
can safely annotate that any trace event may acquire trace.lock even
if dynamically it turns out a particular event doesn't need to flush
and acquire this lock.

This reveals a new edge where we can trace while holding the mheap
lock, so we add this to the lock graph.

For #53789.
Updates #53979.

Change-Id: I13e2f6cd1b621cca4bed0cc13ef12e64d05c89a7
Reviewed-on: https://go-review.googlesource.com/c/go/+/418720
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Run-TryBot: Austin Clements <austin@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>