cmd/trace/v2: goroutine analysis page doesn't identify goroutines consistently #65574
Comments
|
Change https://go.dev/cl/562455 mentions this issue: |
'Doh. Thank you. Yes, these are both issues we should fix. I think using the name for grouping is fine and good.
Yes, this was an oversight that was brought to my attention relatively recently. There might be another issue about it already, but a brief look around didn't reveal anything. I think instead of taking a PC though, we should consider taking the full stack trace. The tracer takes ownership of the goroutine when it reads its status for the case where a goroutine is waiting or in a syscall for an entire generation. We can do the same thing with just a PC, but it's not that much more work to just take the whole stack. |
|
I think we should consider this for backport. I don't think the Go 1.22 cmd/trace tool should be broken in this way forever. :P The backport risk here is very low, because it's only dev tooling. I do think we should maybe open up a separate issue for the "some goroutines are missing a stack" problem. |
|
@gopherbot Please open backport issues for Go 1.21 and Go 1.22. This makes the cmd/trace UI harder to interpret with no easy workaround. The fix is very safe because it only touches debug tooling, specifically EDIT: Go 1.21 backport not necessary. That was muscle memory on my part. |
|
Backport issue(s) opened: #65576 (for 1.21), #65577 (for 1.22). Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases. |
dr2chase
added
the
NeedsFix
Done :) #65634 |
|
Change https://go.dev/cl/562558 mentions this issue: |
…ines To determine the identity of a goroutine for displaying in the trace UI, we should use the root frame from a call stack. This will be the starting function for the goroutine and is the same for each call stack from a given goroutine. The new tracer no longer includes starting PCs for goroutines which existed at the start of tracing, so we can't use a PC for grouping together goroutines any more. Instead, we just use the name of the entry function for grouping. Fixes golang#65574 Change-Id: I5324653316f1acf0ab90c30680f181060ea45dd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/562455 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com>
…e for identifying goroutines To determine the identity of a goroutine for displaying in the trace UI, we should use the root frame from a call stack. This will be the starting function for the goroutine and is the same for each call stack from a given goroutine. The new tracer no longer includes starting PCs for goroutines which existed at the start of tracing, so we can't use a PC for grouping together goroutines any more. Instead, we just use the name of the entry function for grouping. For #65574 Fixes #65577 Change-Id: I5324653316f1acf0ab90c30680f181060ea45dd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/562455 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> (cherry picked from commit a517131) Reviewed-on: https://go-review.googlesource.com/c/go/+/562558 Reviewed-by: Michael Pratt <mpratt@google.com>
…e for identifying goroutines To determine the identity of a goroutine for displaying in the trace UI, we should use the root frame from a call stack. This will be the starting function for the goroutine and is the same for each call stack from a given goroutine. The new tracer no longer includes starting PCs for goroutines which existed at the start of tracing, so we can't use a PC for grouping together goroutines any more. Instead, we just use the name of the entry function for grouping. For golang#65574 Fixes golang#65577 Change-Id: I5324653316f1acf0ab90c30680f181060ea45dd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/562455 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> (cherry picked from commit a517131) Reviewed-on: https://go-review.googlesource.com/c/go/+/562558 Reviewed-by: Michael Pratt <mpratt@google.com>
…e for identifying goroutines To determine the identity of a goroutine for displaying in the trace UI, we should use the root frame from a call stack. This will be the starting function for the goroutine and is the same for each call stack from a given goroutine. The new tracer no longer includes starting PCs for goroutines which existed at the start of tracing, so we can't use a PC for grouping together goroutines any more. Instead, we just use the name of the entry function for grouping. For golang#65574 Fixes golang#65577 Change-Id: I5324653316f1acf0ab90c30680f181060ea45dd7 Reviewed-on: https://go-review.googlesource.com/c/go/+/562455 Reviewed-by: Michael Knyszek <mknyszek@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com> Reviewed-by: David Chase <drchase@google.com> (cherry picked from commit a517131) Reviewed-on: https://go-review.googlesource.com/c/go/+/562558 Reviewed-by: Michael Pratt <mpratt@google.com>
Go version
go version go1.22rc2 darwin/arm64
Output of
go envin your module/workspace:What did you do?
Collected an execution trace with the new v2 tracer and viewed it with
go tool trace. I then navigated to the "Goroutine analysis" page.What did you see happen?
I didn't see the goroutines I was looking for on the analysis page, like
main.main. Many of the names in the "Start location" column look like where a goroutine would be waiting, not where it started. Here's an example of what I see, looking an an execution trace from running unit tests:Here, for example, the goroutines called
runtime.goparkare actually theruntime.gcBgMarkWorkergoroutines, andruntime.chanrecv1is actuallymain.mainwaiting on a channel.What did you expect to see?
The "start location" of a goroutine should be the name of the entry function. For example, the goroutine created by the statement
go myFunction()should be calledmyFunction.The immediate issue is that the current goroutine summary code uses the wrong end of the call stack to identify a goroutine here. There's a secondary issue, which is that the UI uses a PC as a key for grouping together goroutines, but the tracer no longer records the "starting" PC for goroutines which existed at the start of the trace. So grouping by PC is no longer consistent. We could remove the dependency on the starting PC from the goroutine pages and just use the name for grouping. I can send a CL if we're okay with that.
As a followup, should we include starting PCs in traces again? The old tracer recorded the starting PC of every goroutine at the beginning of the trace. The starting PC would mainly be helpful for identifying goroutines where we don't see any other call stacks. We could also consider recording the full call stack of a goroutine the first time we see it, e.g. where it's blocked at the start of the trace, instead of/in addition to the starting PC. Happy to open a separate issue or leave a comment somewhere else if this point is worth discussing more?
cc @mknyszek
The text was updated successfully, but these errors were encountered: