-
Notifications
You must be signed in to change notification settings - Fork 17.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
x/build: misc/reboot.TestRepeatBootstrap flaky during toolchain3 on linux-ppc64le-* #35233
Comments
Can the test be modified to print out more information about the failure?
Maybe stderr has more information. Exit status 2 is not enough.
Who could be sending a kill signal, could that due to timing out?
I'll see if I can reproduce the problem.
…On Tue, Oct 29, 2019 at 8:04 AM Bryan C. Mills ***@***.***> wrote:
The linux-ppc64le builders are frequently failing in TestRepeatBootstrap.
The failure mode seems to always be a compile error when building
cmd/compile/internal/ssa during the Building Go toolchain3 step.
Sometimes the compile command fails with exit status 1 (
https://build.golang.org/log/91e2d3f5fd454609faf1f004a756382ad5aa6e20).
Sometimes it instead fails with signal: killed (
https://build.golang.org/log/00d4e0a84ece047eaa53251c7c4fc0955bba8eea).
Is it possible that this step is running out of RAM?
CC @randall77 <https://github.com/randall77> @bradfitz
<https://github.com/bradfitz> @dmitshur <https://github.com/dmitshur>
@laboger <https://github.com/laboger>
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#35233?email_source=notifications&email_token=ACH7BDCNG3XLAHJ4ZKOGZBDQRAYFJA5CNFSM4JGIGKRKYY3PNVWWK3TUL52HS4DFUVEXG43VMWVGG33NNVSW45C7NFSM4HVBQLYA>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ACH7BDF47DCPW2AZH5XRCWTQRAYFJANCNFSM4JGIGKRA>
.
|
Change https://golang.org/cl/204038 mentions this issue: |
It's already redirecting stdout and stderr to its own stdout/stderr: cmd := exec.Command(filepath.Join(runtime.GOROOT(), "src", makeScript))
cmd.Dir = gorootSrc
cmd.Env = append(os.Environ(), "GOROOT=", "GOROOT_BOOTSTRAP="+runtime.GOROOT())
cmd.Stderr = os.Stderr
cmd.Stdout = os.Stdout
if err := cmd.Run(); err != nil {
t.Fatal(err)
} So right above the "Exit status 2" is how it failed, the lines that @bcmills quoted. Not sure why reboot would take more memory, though. It already did a make.bash successfully if we made it this far, and dist itself doesn't take much memory. Oh---- rundockerbuildlet gives it a tmpfs as its /workdir: And reboot runs at the end, when its /workdir is most full. And because tmpfs, being in memory, contributes to its container memory limit, it has much less memory to work with in this second make.bash later. |
Updates golang/go#35233 Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc Reviewed-on: https://go-review.googlesource.com/c/build/+/204038 Reviewed-by: Bryan C. Mills <bcmills@google.com>
I thought there would be more information about the failure than just "exit status 2". This test appears to be rebuilding the toolchain using the toolchain it just built as its bootstrap compiler. Doesn't that mean it will be changing/updating components of the toolchain as it is using those components for the new build?
Here it is using /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/compile to compile with but the result is an install of cmd/compile. |
Updates golang/go#35233 Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc Reviewed-on: https://go-review.googlesource.com/c/build/+/204038 Reviewed-by: Bryan C. Mills <bcmills@google.com>
Change https://golang.org/cl/366538 mentions this issue: |
The
linux-ppc64le
builders are frequently failing inTestRepeatBootstrap
.The failure mode seems to always be a
compile
error when buildingcmd/compile/internal/ssa
during theBuilding Go toolchain3
step.Sometimes the
compile
command fails withexit status 1
(https://build.golang.org/log/91e2d3f5fd454609faf1f004a756382ad5aa6e20).Sometimes it instead fails with
signal: killed
(https://build.golang.org/log/00d4e0a84ece047eaa53251c7c4fc0955bba8eea).Is it possible that this step is running out of RAM?
CC @randall77 @bradfitz @dmitshur @laboger
The text was updated successfully, but these errors were encountered: