runtime: deadlock when running concurrent builds on MacOS #59657
Comments
gopherbot
added
the
compiler/runtime
|
I tried |
|
This seems to be a problem with Bazel building a helper binary. That seems like a problem that needs to be fixed in Bazel. It's not clear to me that there is anything that the Go project can change to fix this. |
|
Hey Ian, i think that's miss characterizing the issue. Bazel was mentioned to provide the context that multiple similar "go build" commands were being executed in parallel by a build tool. I don't think Bazel is causing the OS-level lock here. I suspect the root cause is multiple "go build" commands raced against the same kevent api. |
Here is the effective command that's being executed in parallel by Bazel. The only difference between them would be the output path |
|
I don't think that Bazel calls |
|
This is the relevant code from rules_go which will generate the command I shared above. https://github.com/bazelbuild/rules_go/blob/3125d8f0971bee56f069bbb33beb6686109fb71d/go/private/rules/binary.bzl#L451-L463 Here we are compiling the |
|
OK, and it seems to me that the deadlock occurs when running the |
|
@ianlancetaylor the deadlock does indeed occurs during "building the You could double check my screenshots to see that the stuck process trees are composed of |
|
Again, this bug would be much simpler to understand if you used plain text rather than screenshots. What I see in the first screenshot is "9 actions running" and then 8 lines starting with "GoToolchainBinaryBuild external/go_sdk/builder [for tool]". Nothing in that tells me that it is running "go build" to build the The other screenshots are inconclusive. The Bazel I'm pretty sure that there is an option you can specify to Bazel that tells it to print out the commands that it executes; maybe it is Let me add that while of course it is possible that there is a bug in |
Hey Ian, I do apologise for the lack of text. The above is the |
mknyszek
added
the
NeedsInvestigation
|
Thanks. You're quite right, it does seem to be running The stack traces you show above seem to show cmd/go waiting for a subcommand to complete. Based on the process tree it is likely waiting for cmd/compile to compile something. What does |
It's because of different Bazel's build configurations, generated from a combination of multiple transitions augmented the Bazel buildgraph. So each action is executed multiple times, once for each unique build configuration.
Im a bit busy until the end of this week so I will provide a reproduce with top later. But from my original issue description, you could review the sampling result of the Note: Since this issue was created, we have merged bazelbuild/rules_go#3536 in rules_go to limit the concurrency of the |
|
The |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes, this could be reproduce on
1.20.3but cannot be reproduce on1.19.8What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
In Bazel's rules_go, we need to compile a helper binary using
go build ...before building anything else.However, due to different Bazel configurations existing in the same repo (i.e. CGO enable/disable), this same helper binary is built multiple times, once for each unique set of configurations.
When trying to build a test in rules_go with
bazel build //tests/core/go_path:copy_path, the helper binary was scheduled by Bazel to be compiled 9 times in parallel. This resulted in 6 differentgo buildprocesses running in parallel at once.Each of these build process could spawn multiple
compilesubprocess as a resultWhat did you expect to see?
Build to complete without issue
What did you see instead?
All these builds are stuck in a deadlock that completely freeze applications running on the same machine. Opening a new Chrome tab, or running
git configwould never finish.Using Activity Monitor's sampling feature we could see call graphs like such for each
go buildprocessIt seems like there are 3 types of stack relating to
libsystem_kernel.dylib / libsystem_pthread.dylibThe last stack seems to be waiting for some event that never finishes (?).
This bug could be reproduced consistently for me with go
1.20.2and1.20.3. It's a bit painful though as I would need to restart my laptop each time. It would go away if I try to build the binary sequentially by limiting Bazel's parallelism, or if I were to downgrade the go version to1.19.8.Here is some additional system info
The text was updated successfully, but these errors were encountered: