Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/tools: document minimum requirements for tests #33986

Open
dmitshur opened this issue Aug 30, 2019 · 7 comments
Open

x/tools: document minimum requirements for tests #33986

dmitshur opened this issue Aug 30, 2019 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) Documentation NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Milestone

Comments

@dmitshur
Copy link
Contributor

Some of the x/tools tests are resource intensive and doing go test golang.org/x/tools/... on a machine with low RAM may fail.

I think it'd be helpful to document the RAM requirement. That way, if the tests run out of memory on a machine that meets the minimum requirements, it becomes easy to determine that the problem lies with the tests and not the machine. The responsibility division is unclear when the requirements are unspecified.

This is similar to issue #19046 but for x/tools.

/cc @ianthehat @heschik @stamblerre @bradfitz

@dmitshur dmitshur added Testing An issue that has been verified to require only test changes, not just a test failure. Builders x/build issues (builders, bots, dashboards) labels Aug 30, 2019
@dmitshur dmitshur added this to the Unreleased milestone Aug 30, 2019
@gopherbot
Copy link

Change https://golang.org/cl/192679 mentions this issue: dashboard: change linux-amd64-longtest machine to n1-highcpu-16

gopherbot pushed a commit to golang/build that referenced this issue Aug 30, 2019
CL 167638 made a change to add more CPU resources to the longtest
builder, with the intention of going from 4 vCPUs, 15 GB RAM to
16 vCPUs, 14.4 GB RAM.

It used n1-highcpu-8 GCE machine type, which actually has 8 vCPUs
and 7.2 GB RAM.¹ Having less RAM than before wasn't the intention.

Fix that by changing n1-highcpu-8 to n1-highcpu-16, which matches
the comment.

¹ https://cloud.google.com/compute/docs/machine-types

Updates golang/go#32831
Updates golang/go#33986
Updates golang/go#25886

Change-Id: I8426867fe33b3bf86576cb13d0d6113cd87f30c1
Reviewed-on: https://go-review.googlesource.com/c/build/+/192679
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@katiehockman katiehockman added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 3, 2019
@katiehockman katiehockman assigned dmitshur and unassigned dmitshur Sep 3, 2019
@stamblerre stamblerre added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 9, 2019
codebien pushed a commit to codebien/build that referenced this issue Nov 13, 2019
CL 167638 made a change to add more CPU resources to the longtest
builder, with the intention of going from 4 vCPUs, 15 GB RAM to
16 vCPUs, 14.4 GB RAM.

It used n1-highcpu-8 GCE machine type, which actually has 8 vCPUs
and 7.2 GB RAM.¹ Having less RAM than before wasn't the intention.

Fix that by changing n1-highcpu-8 to n1-highcpu-16, which matches
the comment.

¹ https://cloud.google.com/compute/docs/machine-types

Updates golang/go#32831
Updates golang/go#33986
Updates golang/go#25886

Change-Id: I8426867fe33b3bf86576cb13d0d6113cd87f30c1
Reviewed-on: https://go-review.googlesource.com/c/build/+/192679
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/213557 mentions this issue: dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM

gopherbot pushed a commit to golang/build that referenced this issue Jan 7, 2020
Start using n1-highcpu-16 machine type instead of n1-highcpu-8
for windows-amd64-longtest and windows-amd64-race builders.

The primary motivation of this change is to increase amount of
RAM available on the race builder from 7.2 GB to 14.4 GB.
CL 208500 has helped by going from 3.6 to 7.2, but empirically
8 GB is not enough for all tests to pass, while 16 GB can be.
We already use 16 GB for some other race and/or long builders.

Updates golang/go#35186
Updates golang/go#33951
Updates golang/go#33986

Change-Id: I8031a517e6e44ce5e5faba085e7c781a1fcab92e
Reviewed-on: https://go-review.googlesource.com/c/build/+/213557
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/214433 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 7.2 GB RAM

gopherbot pushed a commit to golang/build that referenced this issue Jan 14, 2020
Start using n1-highcpu-8 machine type instead of n1-highcpu-4
for the freebsd-amd64-race builder.

The freebsd-amd64-race builder has produced good test results
for the x/tools repo for a long time, but by now it has started
to consistently fail for reasons that seem connected to it having
only 3.6 GB memory. The Windows race builders needed to be bumped
from 7.2 GB to 14.4 GB to run successfully, so this change makes
a small incremental step to bring freebsd-amd64-race closer in
line with other builders. If memory-related problems continue to
occur with 7.2 GB, the next step will be to go up to 14.4 GB.

The freebsd-amd64-race builder is using an older version of FreeBSD.
We may want to start using a newer one for running tests with -race,
but that should be a separate change so we can see the results of
this change without another confounding variable.

Also update all FreeBSD builders to use https in buildletURLTmpl,
because it's expected to work fine and will be more consistent.

Updates golang/go#36444
Updates golang/go#34621
Updates golang/go#29252
Updates golang/go#33986

Change-Id: Idfcefd1c91bddc9f70ab23e02fcdca54fda9d1ac
Reviewed-on: https://go-review.googlesource.com/c/build/+/214433
Run-TryBot: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
@gopherbot

This comment has been minimized.

@gopherbot
Copy link

Change https://golang.org/cl/227859 mentions this issue: dashboard: upsize freebsd-amd64-race builder to 16 vCPUs, 14.4 GB RAM

gopherbot pushed a commit to golang/build that referenced this issue Apr 10, 2020
This is a followup to CL 214433. Start using n1-highcpu-16 machine type
instead of n1-highcpu-8 for the freebsd-amd64-race builder.

Increasing the RAM from 3.6 GB to 7.2 GB has helped golang/go#36444
significantly: the builder stopped failing consistently on x/tools
and resulted in many data races being uncovered in golang/go#36605.

However, by now, it has started to fail consistently again. This
time it seems to be due to low performance, causing the tests in
golang.org/x/tools/internal/lsp/regtest package to fail due with
"context deadline exceeded" errors.

FreeBSD is one of the ports that stays visible when "show only first-
class ports" is checked on build.golang.org. The other -race builders
have all been upgraded to the n1-highcpu-16 machine type by now out
of necessity.

It seems fair to provide the FreeBSD port with an equal amount of
resources, even if the increased memory isn't strictly required yet.
Once this change is applied, if the failures persist, we can be more
confident that the problem is due to the code or the port, rather
than due to this -race builder having 2𝗑 less CPU and RAM resources
compared to other -race builders.

An alternative is to increase timeout for this builder type, but I'm
opting to defer exploring that after equalizing the machine type.

For golang/go#36444.
For golang/go#34621.
For golang/go#29252.
For golang/go#33986.

Change-Id: I41f149365128c7bc6f576c778ac07618acc04612
Reviewed-on: https://go-review.googlesource.com/c/build/+/227859
Reviewed-by: Alexander Rakoczy <alex@golang.org>
@dmitshur
Copy link
Contributor Author

The pattern that seems to establish itself here is that builders that run x/tools tests need approximately 16~ GB RAM. Going with as little as 4 or 8 GB RAM has proven to be insufficient in a number of cases.

I suggest documenting the 16 GB RAM requirement to resolve this issue.

@gopherbot
Copy link

Change https://golang.org/cl/355669 mentions this issue: dashboard: upsize openbsd-386-64 to 16 GB RAM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) Documentation NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure. Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Development

No branches or pull requests

4 participants