I'm not sure whether this problem is getting worse, or we're just having a run of bad luck, but we've got five of what appear to be OOM failures in a row now, four of them on https://golang.org/cl/184165.

(https://build.golang.org/?repo=golang.org%2fx%2ftools#short)

CC @matloob @heschik

Definitely not just a run of bad luck. x/tools went from mostly passing on windows-amd64-race to mostly failing at some point around CL 192277 or CL 184165. I'm guessing that one of those changes increased the memory footprint of the test above some critical threshold.

These also occur regularly on the windows-amd64-longtest builder: https://build.golang.org/log/4837bb3076d9cee69a232dba45f4e4a9a939cf03

Okay. Can the builders' sizes be increased?

Looks like they are currently n1-highcpu-4 ,¹ ² which is only 3.6 GB per instance, so probably they should be made substantially larger.

@dmitshur upsized the corresponding linux builders to n1-highcpu-16 in CL 192679.

(CC @bradfitz @toothrot @cagedmantis)

¹https://github.com/golang/build/blob/0babffaf2095de16932bb07c70d58a2730a161fd/dashboard/builders.go#L2029-L2058
²https://github.com/golang/build/blob/0babffaf2095de16932bb07c70d58a2730a161fd/dashboard/builders.go#L421-L441

I agree we should increase the memory for the -race builder at the very least. In order to do that, we need to make a decision about the machine type to use next.

Right now all the windows/amd64 hosts use n1-highcpu-4 machine type, since they were added in 2017 in CL 41393. This includes the normal builders, the -race builder, and the -longtest builders.

Highcpu is described as:

High-CPU machine types are ideal for tasks that require a moderate increase of vCPUs relative to memory. High-CPU machine types have 0.90 GB of memory per vCPU.

In contrast to standard:

Standard machine types have 4 GB of memory per vCPU.

So, we can consider changing the windows -race builder to use either n1-highcpu-8, or n1-standard-4. Here's a comparison (based on https://cloud.google.com/compute/docs/machine-types and https://cloud.google.com/compute/vm-instance-pricing):

I would prefer for us to make smaller changes, and iterate based on the results. My current impression is that the standard machine type would be a better fit for our needs, so I propose we change all the windows hosts to use n1-standard-4 (Option 2) instead of the current n1-highcpu-4. Both options seem quite close so I'm on the fence. Feedback welcome. Otherwise we can proceed with that plan as the next step.

I would like to have lower-latency longtest slowbots, so (budget permitting) I have a sight preference for the highcpu configuration.

(But really, anything is better than OOMing at head, so I'm not going to hold things up if folks would prefer a different option.)

Bryan, how does n1-highcpu-8 for -longtest and n1-standard-4 for the rest (normal and -race) sound?

I would be inclined to do n1-highcpu-8 for -longtest and -race and keep n1-highcpu-4 for the rest, although it's possible that -race will need more than 7.2GB to run x/tools.

(If the “short” builders are doing fine today with only 3.6 GB, I'm doubtful that throwing more RAM at them will do much, although I suppose it could improve filesystem buffering performance.)

FWIW, it looks like freebsd-amd64-race is doing fine on n1-highcpu-4, so it may be worth some time for the x/tools folks to figure out why Windows isn't managing. (Are some tests being unexpectedly skipped on FreeBSD? Is the platform just that much more efficient?)

I'm having a hard time following the configuration for linux-amd64-race, but I believe it may be on an n1-standard-4 (due to running in a container).

it may be worth some time for the x/tools folks to figure out why Windows isn't managing. (Are some tests being unexpectedly skipped on FreeBSD? Is the platform just that much more efficient?)

Issue #33986 is relevant for that.

I'm having a hard time following the configuration for linux-amd64-race, but I believe it may be on an n1-standard-4 (due to running in a container).

I believe that is accurate.

I would be inclined to do n1-highcpu-8 for -longtest and -race and keep n1-highcpu-4 for the rest,

This sounds good to me as a small incremental step to try here. Will mail a CL.

Change https://golang.org/cl/208500 mentions this issue: dashboard: upsize machine type for windows-amd64-{longtest,race} builders

Looks like the CL above helped; we got our first near-pass of x/tools on windows-amd64-longtest builder in this recent run. It found #35839.

Change https://golang.org/cl/213557 mentions this issue: dashboard: upsize windows-amd64-{longtest,race} builders to 14.4 GB RAM

FWIW, it looks like freebsd-amd64-race is doing fine on n1-highcpu-4, so it may be worth some time for the x/tools folks to figure out why Windows isn't managing. (Are some tests being unexpectedly skipped on FreeBSD? Is the platform just that much more efficient?)

Those are good questions, but they are hard to answer without a resolution to #33986. I think that issue can be used to track this work.

The freebsd-amd64-race builder is no longer okay with just 3.6 GB, as reported in issue #36444. The current plan there is to increase its size to 7.2 GB, which is still half of what Windows needed to be able to pass (see issue #35186 for details).

I'll close this issue as resolved via CL 213557 because there aren't frequent OOMs in x/tools/internal/lsp on windows-amd64-longtest builder anymore. See #35186 (comment) specifically.