Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: lock ranking failure between sysmon and gscan #54553

Closed
heschi opened this issue Aug 19, 2022 · 6 comments
Closed

runtime: lock ranking failure between sysmon and gscan #54553

heschi opened this issue Aug 19, 2022 · 6 comments
Assignees
@aclements
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone
Go1.20

Comments

@heschi
Copy link
Contributor

heschi commented Aug 19, 2022

2022-08-18T18:18:45-1aa5730/linux-amd64-staticlockranking
2022-08-18T17:26:10-b23d469/linux-amd64-staticlockranking
2022-08-18T17:09:37-c82bbc0/linux-amd64-staticlockranking
2022-08-18T16:35:13-b2c0417/linux-amd64-staticlockranking
2022-08-18T16:15:06-c7c78f1/linux-amd64-staticlockranking
2022-08-18T16:12:31-b8b9e83/linux-amd64-staticlockranking
2022-08-18T14:53:01-741ab7e/linux-amd64-staticlockranking
2022-08-18T05:32:36-a2c2f06/linux-amd64-staticlockranking
2022-08-18T02:51:24-04d3d6b/linux-amd64-staticlockranking
2022-08-18T00:30:19-b30ba3d/linux-amd64-staticlockranking
2022-08-17T23:18:24-9709d92/linux-amd64-staticlockranking
2022-08-17T18:47:33-57d0551/linux-amd64-staticlockranking
2022-08-17T17:40:05-dea67a9/linux-amd64-staticlockranking
2022-08-17T16:48:18-b11b4b4/linux-amd64-staticlockranking
2022-08-17T16:26:22-5b0ce94/linux-amd64-staticlockranking
2022-08-17T04:02:17-c411886/linux-amd64-staticlockranking
2022-08-17T03:24:24-c6be710/linux-amd64-staticlockranking

cc @golang/runtime
@heschi heschi added the NeedsFix The path to resolution is known, but the work has not been done. label Aug 19, 2022
@heschi heschi added this to the Go1.20 milestone Aug 19, 2022
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Aug 19, 2022
@prattmic
Copy link
Member 

207493  ======
0 : sysmon 1 0xd66998
1 : traceBuf 23 0xd7df98
2 : traceStackTab 42 0xd6def8
3 : gscan 34 0x0
fatal error: lock ordering problem

@heschi heschi mentioned this issue Aug 19, 2022
Closed
@aclements
Copy link
Member

The failure is almost always in testing cmd/trace (occasionally runtime/trace). It's relatively easy to reproduce:

$ git checkout c6be710
$ GOEXPERIMENT=staticlockranking ./make.bash && stress2 -max-runs 1000 -- go test -count 1 -short cmd/trace
...
926 passes, 74 fails, avg 2s

Presumably I introduced this when I lowered the trace locks to be near the bottom of the lock ranking, and below stack growth in particular. I made sure that trace.lock was only acquired on the system stack, but I think I missed traceStackTable.lock.

@aclements
Copy link
Member

Confirmed that I was able to reproduce at cc8bac8, but not at cc8bac8~. I'll send a fix shortly.

@gopherbot
Copy link

Change https://go.dev/cl/425097 mentions this issue: runtime: move traceStackTable.lock to the system stack

@bcmills
Copy link
Contributor

bcmills commented Aug 23, 2022

The failure is almost always in testing cmd/trace (occasionally runtime/trace).

Hmm, does that also explain the timeouts reported in #54594?

@aclements
Copy link
Member

Hmm, does that also explain the timeouts reported in #54594?

The first instance of that issue happened before my change, so it doesn't directly relate to this issue. But I'll comment on that issue.

@golang golang locked and limited conversation to collaborators Aug 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@aclements aclements

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Milestone
Go1.20
Development

No branches or pull requests
5 participants
@prattmic @aclements @bcmills @gopherbot @heschi