Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: misc/reboot.TestRepeatBootstrap flaky during toolchain3 on linux-ppc64le-* #35233

Closed
bcmills opened this issue Oct 29, 2019 · 5 comments
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Oct 29, 2019

The linux-ppc64le builders are frequently failing in TestRepeatBootstrap.

The failure mode seems to always be a compile error when building cmd/compile/internal/ssa during the Building Go toolchain3 step.

Sometimes the compile command fails with exit status 1 (https://build.golang.org/log/91e2d3f5fd454609faf1f004a756382ad5aa6e20).
Sometimes it instead fails with signal: killed (https://build.golang.org/log/00d4e0a84ece047eaa53251c7c4fc0955bba8eea).

Is it possible that this step is running out of RAM?

CC @randall77 @bradfitz @dmitshur @laboger

@bcmills bcmills added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-ppc64x labels Oct 29, 2019
@gopherbot gopherbot added this to the Unreleased milestone Oct 29, 2019
@laboger
Copy link
Contributor

laboger commented Oct 29, 2019 via email

@gopherbot
Copy link

Change https://golang.org/cl/204038 mentions this issue: dashboard: skip reboot test on ppc64x builders

@bradfitz
Copy link
Contributor

Can the test be modified to print out more information about the failure?
Maybe stderr has more information.

It's already redirecting stdout and stderr to its own stdout/stderr:

        cmd := exec.Command(filepath.Join(runtime.GOROOT(), "src", makeScript))
        cmd.Dir = gorootSrc
	cmd.Env = append(os.Environ(), "GOROOT=", "GOROOT_BOOTSTRAP="+runtime.GOROOT())
        cmd.Stderr = os.Stderr
        cmd.Stdout = os.Stdout
        if err := cmd.Run(); err != nil {
                t.Fatal(err)
        }

So right above the "Exit status 2" is how it failed, the lines that @bcmills quoted.

Not sure why reboot would take more memory, though. It already did a make.bash successfully if we made it this far, and dist itself doesn't take much memory.

Oh---- rundockerbuildlet gives it a tmpfs as its /workdir:

https://github.com/golang/build/blob/c53d4ba783d113c1e97ca07be915b7e2f6142ede/cmd/rundockerbuildlet/rundockerbuildlet.go#L157

And reboot runs at the end, when its /workdir is most full.

And because tmpfs, being in memory, contributes to its container memory limit, it has much less memory to work with in this second make.bash later.

gopherbot pushed a commit to golang/build that referenced this issue Oct 29, 2019
Updates golang/go#35233

Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc
Reviewed-on: https://go-review.googlesource.com/c/build/+/204038
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@laboger
Copy link
Contributor

laboger commented Oct 30, 2019

I thought there would be more information about the failure than just "exit status 2".

This test appears to be rebuilding the toolchain using the toolchain it just built as its bootstrap compiler. Doesn't that mean it will be changing/updating components of the toolchain as it is using those components for the new build?

go build cmd/compile/internal/ssa: /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/compile: exit status 1
go tool dist: FAILED: /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/go_bootstrap install -gcflags=all= -ldflags=all= -a -i cmd/asm cmd/cgo cmd/compile cmd/link: exit status 1
--- FAIL: TestRepeatBootstrap (100.86s)
    reboot_test.go:50: exit status 2

Here it is using /workdir/tmp/reboot-goroot202913779/pkg/tool/linux_ppc64le/compile to compile with but the result is an install of cmd/compile.

@bradfitz bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Nov 4, 2019
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 4, 2019
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
Updates golang/go#35233

Change-Id: Icfa19da5d0dd3c47ffa94d11f069b8836c86cdcc
Reviewed-on: https://go-review.googlesource.com/c/build/+/204038
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/366538 mentions this issue: misc/reboot: skip TestRepeatBootstrap on short builders

@golang golang locked and limited conversation to collaborators Nov 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-ppc64x Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

4 participants