Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/cgo/internal/testsanitizers,x/build: LUCI clang15 builders failing #65469

Closed
mknyszek opened this issue Feb 2, 2024 · 9 comments
Closed
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone

Comments

@mknyszek
Copy link
Contributor

mknyszek commented Feb 2, 2024

I've been working on bringing up the clang15 builders this week on LUCI, and the ASAN tests are failing because of a failure to symbolize the output. I've narrowed down the root cause to a missing zlib dependency (in hindsight, pretty obvious from the output) in our image.

Filing an issue to track fixing this. It's especially important because these builders are run in an advisory manner as part of the release.

@mknyszek mknyszek self-assigned this Feb 2, 2024
@mknyszek mknyszek added this to the Unreleased milestone Feb 2, 2024
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Feb 2, 2024
@gopherbot
Copy link

Change https://go.dev/cl/561595 mentions this issue: env/linux-x86-bullseye: add zlib1g to dependencies to install

@dmitshur dmitshur added the NeedsFix The path to resolution is known, but the work has not been done. label Feb 5, 2024
@mknyszek
Copy link
Contributor Author

mknyszek commented Feb 5, 2024

This will take some time to propagate. I suspect it should look OK by tomorrow.

@mknyszek
Copy link
Contributor Author

mknyszek commented Feb 6, 2024

I added zlib to the image but it wasn't enough. I tracked down the problem to the fact that the official LLVM project's release of clang does NOT build llvm-symbolizer with zlib support.

This is unfortunate. The only alternative is to use a different clang. The nice thing about the current system I set up for LUCI is that clang is using just the regular official LLVM release maintained easily in CIPD, so it's easy to update, change, and manage.

A few available options (all unfortunate) are:

  • Use the LLVM distro-packaged version of clang (they build llvm-symbolizer against zlib) and bake it into the image. (From https://apt.llvm.org, so at least it's official.)
    • This is unfortunate because we have to have a separate set of machines that contain clang, or we add clang to every linux-amd64 machine.
    • It makes updating clang harder, since clang isn't available in Debian by default. The package appears to need to be manually added.
  • Use Chromium's clang build, which likely (though I haven't confirmed) has it.
    • It feels wrong to pin ourselves to a custom clang build that also isn't owned by us.
  • Make the process for adding a new clang build involve building clang ourselves.
    • This seems overly onerous and building against a (technically) custom clang seems like the wrong call.

However, there's a different direction we can take.

The ASAN tests already skip checking for file and line numbers (symbolized results) in the output on old GCC versions. However, AFAICT, the version of GCC on our builders does actually symbolize the output. I think the primary purpose of these tests is to make sure our debug information is working correctly, not necessarily integration with specific toolchains. So, theoretically, we have coverage on our debug information working properly already.

What I propose:

  1. File a bug against upstream LLVM to build against zlib in their releases.
  2. Skip the symbolization part of the test for clang.
  3. File a bug to track turning this part of the test back on once LLVM upstream (hopefully) starts building their releases against zlib.

Based on the thread I found this request seems reasonable to the LLVM maintainers. I'll note that apparently the exact LLVM configuration used in the release is up to whoever is doing the release anyway, so my assumption is that there are already some inconsistencies in their process, and requiring zlib probably wouldn't be too hard on users of clang. But I don't know.

@gopherbot
Copy link

Change https://go.dev/cl/562675 mentions this issue: cmd/cgo/internal/testsanitizers: disable location checking for clang

@gopherbot
Copy link

Change https://go.dev/cl/562538 mentions this issue: env/linux-x86-bullseye: add clang to image

@mknyszek
Copy link
Contributor Author

mknyszek commented Feb 9, 2024

@gopherbot Please open backport issues for Go 1.21 and Go 1.22. The LUCI clang builder on those branches is currently broken, and we should cherry-pick the fix applied here.

@gopherbot
Copy link

Backport issue(s) opened: #65640 (for 1.21), #65641 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

@mknyszek mknyszek changed the title x/build: LUCI clang15 builders failing cmd/cgo/internal/testsanitizers,x/build: LUCI clang15 builders failing Feb 9, 2024
@dmitshur dmitshur added the Testing An issue that has been verified to require only test changes, not just a test failure. label Feb 9, 2024
@gopherbot
Copy link

Change https://go.dev/cl/563015 mentions this issue: [release-branch.go1.22] cmd/cgo/internal/testsanitizers: disable location checking for clang

@gopherbot
Copy link

Change https://go.dev/cl/562999 mentions this issue: [release-branch.go1.21] cmd/cgo/internal/testsanitizers: disable location checking for clang

gopherbot pushed a commit that referenced this issue Feb 16, 2024
…tion checking for clang

Pending a resolution to #65606, this CL marks clang's ASAN runtime as
unable to symbolize stack traces to unblock the LUCI clang builder.

For #65606.
For #65469.
Fixes #65640.

Change-Id: I649773085aff30e5703e7f7ac2c72a0430a015c2
Cq-Include-Trybots: luci.golang.try:go1.21-linux-amd64-clang15
Reviewed-on: https://go-review.googlesource.com/c/go/+/562675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(cherry picked from commit d94ab59)
Reviewed-on: https://go-review.googlesource.com/c/go/+/562999
gopherbot pushed a commit that referenced this issue Feb 16, 2024
…tion checking for clang

Pending a resolution to #65606, this CL marks clang's ASAN runtime as
unable to symbolize stack traces to unblock the LUCI clang builder.

For #65606.
For #65469.
Fixes #65641.

Change-Id: I649773085aff30e5703e7f7ac2c72a0430a015c2
Cq-Include-Trybots: luci.golang.try:go1.22-linux-amd64-clang15
Reviewed-on: https://go-review.googlesource.com/c/go/+/562675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(cherry picked from commit d94ab59)
Reviewed-on: https://go-review.googlesource.com/c/go/+/563015
ezz-no pushed a commit to ezz-no/go-ezzno that referenced this issue Feb 18, 2024
Pending a resolution to golang#65606, this CL marks clang's ASAN runtime as
unable to symbolize stack traces to unblock the LUCI clang builder.

For golang#65606.
Fixes golang#65469.

Change-Id: I649773085aff30e5703e7f7ac2c72a0430a015c2
Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-clang15
Reviewed-on: https://go-review.googlesource.com/c/go/+/562675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
bradfitz pushed a commit to tailscale/go that referenced this issue Mar 5, 2024
…tion checking for clang

Pending a resolution to golang#65606, this CL marks clang's ASAN runtime as
unable to symbolize stack traces to unblock the LUCI clang builder.

For golang#65606.
For golang#65469.
Fixes golang#65641.

Change-Id: I649773085aff30e5703e7f7ac2c72a0430a015c2
Cq-Include-Trybots: luci.golang.try:go1.22-linux-amd64-clang15
Reviewed-on: https://go-review.googlesource.com/c/go/+/562675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(cherry picked from commit d94ab59)
Reviewed-on: https://go-review.googlesource.com/c/go/+/563015
romaindoumenc pushed a commit to TroutSoftware/go that referenced this issue Mar 6, 2024
…tion checking for clang

Pending a resolution to golang#65606, this CL marks clang's ASAN runtime as
unable to symbolize stack traces to unblock the LUCI clang builder.

For golang#65606.
For golang#65469.
Fixes golang#65641.

Change-Id: I649773085aff30e5703e7f7ac2c72a0430a015c2
Cq-Include-Trybots: luci.golang.try:go1.22-linux-amd64-clang15
Reviewed-on: https://go-review.googlesource.com/c/go/+/562675
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
(cherry picked from commit d94ab59)
Reviewed-on: https://go-review.googlesource.com/c/go/+/563015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
Status: Done
Development

No branches or pull requests

3 participants