Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build: collect core dumps from builders #49165

Open
prattmic opened this issue Oct 26, 2021 · 2 comments
Open

x/build: collect core dumps from builders #49165

prattmic opened this issue Oct 26, 2021 · 2 comments
Labels
Builders x/build issues (builders, bots, dashboards) FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@prattmic
Copy link
Member

Sometimes tests dump core, perhaps with no extra crash information. Investigating these issues would be much improved if we could get the core file from the builder.

This came up recently in https://golang.org/cl/358675, where https://storage.googleapis.com/go-build-log/2fb3aaf5/linux-amd64_26aedb56.log failed with little more than:

signal: segmentation fault (core dumped)
FAIL	runtime	5.189s
FAIL

#35248 is another case where post-submit failures could likely have been fixed without a reproducer if the core dump was available.

I believe there is an issue regarding collecting multiple files from builders generally which is likely a prerequisite for this, though I can't find it...

cc @mknyszek @golang/release
@prattmic prattmic added the Builders x/build issues (builders, bots, dashboards) label Oct 26, 2021
@gopherbot gopherbot added this to the Unreleased milestone Oct 26, 2021
@mknyszek
Copy link
Contributor

Looks like this needs:

  1. The ability to collect multiple files from a builder,
  2. setting a unique core pattern in run.bash,
  3. enabling core dumps in run.bash (currently disabled),
  4. running builders with GOTRACEBACK=crash instead of GOTRACEBACK=system.

Then the coordinator could, say, blindly search the work tree and /tmp for anything resembling the core dump pattern and then upload it to some GCS bucket with an expiry (so it doesn't grow infinitely large).

For most use-cases, it's probably enough to stop there. A stretch goal would be surfacing it in the UI somehow.

@dmitshur dmitshur added FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Oct 26, 2021
@prattmic prattmic mentioned this issue Feb 24, 2022
Open
@bcmills bcmills mentioned this issue Jun 22, 2022
Closed
@thanm
Copy link
Contributor

thanm commented Jan 31, 2023

Another similar use case would be copying back profiles (coverage profiles, for example, or CPU profiles for PGO purposes).

@bcmills bcmills mentioned this issue Feb 1, 2023
Open
@mknyszek mknyszek mentioned this issue Jul 5, 2023
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Release
Status: Planned
Milestone
Unreleased
Development

No branches or pull requests
5 participants
@mknyszek @prattmic @dmitshur @gopherbot @thanm