Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: freebsd-amd64-race logs for x/tools marked as failing, but do not include error messages #49347

Closed
bcmills opened this issue Nov 4, 2021 · 7 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 4, 2021

(Note that the below regexp is not specific to freebsd or to x/tools. It is just looking for logs that contain only non-error lines.)

greplogs --dashboard -md -l -e '(?m)\A[^ ]+ at.*\n\n(?:(?:[:][:] Running .+\n\n)|(?:go: downloading [^ ]+ v[^ ]+\n)|(?:ok\s+[^ ]+\s+[^ ]+s\n)|(?:[?]\s+[^ ]+\s+\[no test files\]\n))+\z'

2021-11-04T07:05:31-84e69e7-2622235/freebsd-amd64-race
2021-11-03T23:49:25-84e69e7-fa19146/freebsd-amd64-race
2021-11-03T19:32:33-3b1a674-74f99d0/freebsd-amd64-race
2021-11-03T00:49:50-714668c-a4b2c57/freebsd-amd64-race
2021-11-02T23:44:58-714668c-a0f373c/freebsd-amd64-race
2021-11-02T16:40:04-058ed05-8c8baad/freebsd-amd64-race
2021-11-02T00:12:17-ee08195-81fea0b/freebsd-amd64-race
2021-10-28T16:54:58-c8ad2e1-6bd0e7f/freebsd-amd64-race
2021-10-26T01:18:46-f916b54-49fda9e/freebsd-amd64-race
2021-10-25T14:51:39-fc8b4ca-1ed060c/freebsd-amd64-race
2021-10-17T02:23:16-98f6e03-fa7d11a/freebsd-amd64-race
2021-10-16T16:27:40-98f6e03-640a49b/freebsd-amd64-race
2021-10-16T14:50:21-98f6e03-680caf1/freebsd-amd64-race
2021-10-15T20:34:15-98f6e03-1b072b3/freebsd-amd64-race
2021-10-15T14:09:01-98f6e03-1cbec68/freebsd-amd64-race
2021-10-15T02:45:22-288d0b3-2ac3bdf/freebsd-amd64-race
2021-10-15T02:02:50-ce04ca3-3da0ff8/freebsd-amd64-race
2021-10-15T01:35:56-ce04ca3-0c45ed0/freebsd-amd64-race
2021-10-14T23:32:09-3a269dc-9e8ed86/freebsd-amd64-race
2021-10-14T18:44:37-e69ba9d-a37bebc/freebsd-amd64-race

CC @golang/release

@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 4, 2021
@gopherbot gopherbot added this to the Unreleased milestone Nov 4, 2021
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD labels Nov 4, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Nov 11, 2021

I think the process may be being killed due to memory exhaustion.

The freebsd-amd64-race builder appears to be running on an e2-highcpu-16 builder, while the normal linux-amd64-race builder is running on an e2-standard-4. (https://play.golang.org/p/5LXUgwHEZPn)

According to https://cloud.google.com/compute/vm-instance-pricing, e2-standard-4 has 4 CPUs and 16GB memory, while e2-highcpu-16 has 16 CPUs and 16GB.

Since the default GOMAXPROCS is based on the CPU count, and the default test parallelism is based on GOMAXPROCS, I suspect that the increased CPU-to-RAM ratio on the highcpu host is causing it to run too many gopls tests in parallel and run out of RAM.

@findleyr, @golang/release: should we address this by adjusting the host shape, or by adding some kind of additional limiting to the test, or explicitly setting GOMAXPROCS on this builder (which would possibly reduce its testing throughput), or something else entirely?

@findleyr
Copy link
Contributor

findleyr commented Dec 3, 2021

Comparing these two recent builds:
Failure: https://build.golang.org/log/4d807f4a6b0659eea741ece539bd6a39a4fc0c1a
Success: https://build.golang.org/log/2d9892c7aaf14942405fb6b82ae24234b1807bec

We can see that the failure is not finishing all x/tools tests -- the last test reported is go/loader.
The next test in the success is go/packages, so I suspect this is failing in go/packages (@bcmills is that a viable conclusion?)

@findleyr
Copy link
Contributor

findleyr commented Dec 3, 2021

If this is OOMing in go/packages (which we suspect may lead to this type of silent failure), then I can think of two possible sources:

Looking specifically at some of the go/packages tests, I am also curious about why the tests are so slow and memory intensive, based on what they are doing. e.g. TestLoadDifferentPatterns takes 3s, but that seems extremely slow based on my understanding of our type-checking speed and the scope of that test.

@bcmills
Copy link
Contributor Author

bcmills commented Dec 3, 2021

The next test in the success is go/packages, so I suspect this is failing in go/packages (@bcmills is that a viable conclusion?)

I think so — well spotted!

I can think of two possible sources:

That is an interesting theory! But that change was merged in July, and the first of these failures started in October.

My leading theory is still machine shape (this builder has a very high CPU-to-RAM ratio), but either or both of those could be aggravating factors as well.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 31, 2022

This builder is still failing multiple times per day on x/tools in multiple different packages. We could theoretically fix the tests for each of those packages to scale by RAM, but at this point I'd really just rather make a pointwise change and be done with it.

@golang/release: could we try scaling this builder back to a more balanced configuration?

greplogs --dashboard -md -l -e \\Afreebsd-amd64-race -E . --since=2022-01-28

2022-01-31T20:30:41-492cf3a-41f485b/freebsd-amd64-race
2022-01-31T13:43:52-939c2c0-41f485b/freebsd-amd64-race
2022-01-29T19:58:17-939c2c0-1226547/freebsd-amd64-race
2022-01-29T03:53:30-939c2c0-5fd8c9b/freebsd-amd64-race
2022-01-28T22:17:23-939c2c0-654d5f4/freebsd-amd64-race
2022-01-28T20:07:54-939c2c0-8314544/freebsd-amd64-race

@gopherbot
Copy link

Change https://golang.org/cl/382058 mentions this issue: dashboard: switch host-freebsd-13_0-big to e2-standard-4 machines

gopherbot pushed a commit to golang/build that referenced this issue Jan 31, 2022
This reduces the freebsd-amd64-race builder to 4 vCPUs instead of 16,
which should also reduce the default GOMAXPROCS and default
-test.parallel settings on that builder.

The theory is that by switching to a standard CPU-to-RAM ratio, we
will avoid OOMing in memory-bound x/tools tests, which scale their
parallelism by GOMAXPROCS rather than the machine's total RAM size.

For golang/go#49347

Change-Id: I6960efc5636fef7d2e6197aee33212a7823f0970
Reviewed-on: https://go-review.googlesource.com/c/build/+/382058
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@bcmills bcmills self-assigned this Apr 27, 2022
@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 27, 2022
@bcmills
Copy link
Contributor Author

bcmills commented May 16, 2022

Lots of recent failures on this builder, but — crucially! — none missing error messages. I'm calling this fixed. (We can always reopen — or open a new issue — if that turns out to be wrong.)

greplogs -l -e \\Afreebsd-amd64-race -E . --since=2022-05-01
2022-05-05T18:49:32-983906f/freebsd-amd64-race
2022-05-05T18:49:25-a7ab208/freebsd-amd64-race
2022-05-05T18:49:17-5073c1c/freebsd-amd64-race
2022-05-04T21:35:22-a5481fb-aeb933d/freebsd-amd64-race
2022-05-04T21:03:31-a5481fb-2f23364/freebsd-amd64-race
2022-05-04T20:30:17-a5481fb-e1e056f/freebsd-amd64-race
2022-05-04T20:20:28-a5481fb-fbb47e8/freebsd-amd64-race
2022-05-04T20:06:32-a5481fb-fd6ef06/freebsd-amd64-race
2022-05-04T18:54:31-a5481fb-037b209/freebsd-amd64-race
2022-05-04T18:51:19-1b0f9fb/freebsd-amd64-race
2022-05-04T18:05:18-a5481fb-ffe48e0/freebsd-amd64-race
2022-05-04T18:05:18-78819d0-ffe48e0/freebsd-amd64-race
2022-05-04T16:13:20-f52b4ec/freebsd-amd64-race
2022-05-03T20:54:41-d24c65d/freebsd-amd64-race
2022-05-03T20:49:59-89c0dd8/freebsd-amd64-race
2022-05-03T20:49:46-2c0a988/freebsd-amd64-race
2022-05-03T15:14:09-f01c20b/freebsd-amd64-race
2022-05-03T15:13:53-91f8630/freebsd-amd64-race
2022-05-03T15:13:45-b4d8114/freebsd-amd64-race
2022-05-03T15:13:35-7e4bc74/freebsd-amd64-race
2022-05-03T15:13:21-973dcbb/freebsd-amd64-race
2022-05-03T15:12:52-129dcb7/freebsd-amd64-race
2022-05-03T15:12:45-473c996/freebsd-amd64-race
2022-05-03T15:12:38-375d696/freebsd-amd64-race
2022-05-03T15:12:31-d36d5bd/freebsd-amd64-race
2022-05-03T15:12:21-4649a43/freebsd-amd64-race
2022-05-03T15:12:12-85d4664/freebsd-amd64-race
2022-05-03T15:12:04-986a310/freebsd-amd64-race
2022-05-03T15:11:55-0feebe6/freebsd-amd64-race
2022-05-03T15:11:42-01359b4/freebsd-amd64-race
2022-05-02T21:05:53-04fc2ba-a887579/freebsd-amd64-race

@bcmills bcmills closed this as completed May 16, 2022
@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 16, 2022
@rsc rsc unassigned bcmills Jun 23, 2022
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-FreeBSD
Projects
None yet
Development

No branches or pull requests

3 participants