Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build: add a darwin/amd64 longtest builder #35678

Closed
bcmills opened this issue Nov 18, 2019 · 27 comments
Closed

x/build: add a darwin/amd64 longtest builder #35678

bcmills opened this issue Nov 18, 2019 · 27 comments
Assignees
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-Darwin
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 18, 2019

Now that we have a Windows longtest builder (#26529), I'm wondering where else we might have a coverage gap.

It would be nice to have a darwin builder running non-short tests too. I doubt that it will turn up many regressions for cmd/go in particular, but the system calls are enough different — and the installed base of cmd/go users on macOS is large enough — that I suspect it would have a net positive value.

CC @golang/release

@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. new-builder labels Nov 18, 2019
@gopherbot gopherbot added this to the Unreleased milestone Nov 18, 2019
@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Nov 18, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Nov 18, 2019

(This message brought to you by cmd/go.tooSlow, which probably ought to just Not Be a Thing.)

go/src/cmd/go/go_test.go

Lines 49 to 53 in a23f9af

// In -short mode; skip test, except run it on the {darwin,linux,windows}/amd64 builders.
if testenv.Builder() != "" && runtime.GOARCH == "amd64" && (runtime.GOOS == "linux" || runtime.GOOS == "darwin" || runtime.GOOS == "windows") {
return
}
t.Skip("skipping test in -short mode")

@bcmills
Copy link
Contributor Author

bcmills commented Jan 6, 2020

CC @matloob @jayconrod

@dmitshur dmitshur changed the title x/build: add a darwin longtest builder x/build: add a darwin/amd64 longtest builder Nov 16, 2021
@dmitshur
Copy link
Contributor

I've folded the identical #26530 into this issue. I've also retitled this one to be about darwin/amd64, since #49055 now exists for darwin/arm64.

@bcmills
Copy link
Contributor Author

bcmills commented Mar 7, 2023

This was moved to the Planned project state back in November — do we have an ETA on when it will happen, or is it still in the backlog?

I'd still like to get rid of the builder check in the tooSlow test helper because it fails to detect builders that really are slower (like the darwin-amd64-race builder), such as in #58918 and #58919, and I don't want to add even more complexity to the test to try to hard-code more information about specific builders.

More clarity about when this is likely to happen would help me evaluate the tradeoffs of that change.

@gopherbot
Copy link

Change https://go.dev/cl/474137 mentions this issue: cmd/go: don't run slow tests on non-longtest builders

gopherbot pushed a commit that referenced this issue Mar 7, 2023
Also annotate calls to tooSlow with specific reasons.

This will somewhat reduce test coverage on the 'darwin' builders until
we have darwin 'longtest' builders (#35678,#49055), but still seems
worthwhile to avoid alert fatigue from tests that really shouldn't be
running in the short configurations.

Fixes #58918.
Fixes #58919.

Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313
Reviewed-on: https://go-review.googlesource.com/c/go/+/474137
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/474580 mentions this issue: [release-branch.go1.20] cmd/go: avoid running slow tests on non-longtest builders

@gopherbot
Copy link

Change https://go.dev/cl/474581 mentions this issue: [release-branch.go1.19] cmd/go: avoid running slow tests on non-longtest builders

gopherbot pushed a commit that referenced this issue Mar 9, 2023
…est builders

Also annotate calls to tooSlow with specific reasons.

This will somewhat reduce test coverage on the 'darwin' builders until
we have darwin 'longtest' builders (#35678,#49055), but still seems
worthwhile to avoid alert fatigue from tests that really shouldn't be
running in the short configurations.

Updates #58918.
Updates #58919.
Fixes #58937.

Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313
Reviewed-on: https://go-review.googlesource.com/c/go/+/474137
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit 9f532dd)
Reviewed-on: https://go-review.googlesource.com/c/go/+/474581
gopherbot pushed a commit that referenced this issue Mar 9, 2023
…est builders

Also annotate calls to tooSlow with specific reasons.

This will somewhat reduce test coverage on the 'darwin' builders until
we have darwin 'longtest' builders (#35678,#49055), but still seems
worthwhile to avoid alert fatigue from tests that really shouldn't be
running in the short configurations.

Updates #58918.
Updates #58919.
Fixes #58938.

Change-Id: I0000f0084b262beeec3eca3e9b8a45d61fab4313
Reviewed-on: https://go-review.googlesource.com/c/go/+/474137
Reviewed-by: Ian Lance Taylor <iant@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
(cherry picked from commit 9f532dd)
Reviewed-on: https://go-review.googlesource.com/c/go/+/474580
@heschi
Copy link
Contributor

heschi commented Mar 13, 2023

It's still in the backlog, no particular ETA. Adding the builder obviously isn't too hard, but we'd have to add more machines and that means re-evaluating our budget.

@bcmills
Copy link
Contributor Author

bcmills commented Mar 13, 2023

FWIW, from my perspective the builder would be useful even if it doesn't have enough machines to run at every CL.

(Many of the reverse builders are in that state, and they're still useful for detecting deterministic failures, narrowing down bisection windows, and performing SlowBot runs on particularly risky changes.)

@prattmic prattmic self-assigned this Mar 28, 2023
@gopherbot
Copy link

Change https://go.dev/cl/479837 mentions this issue: dashboard: add darwin-amd64-longtest builder

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 28, 2023
gopherbot pushed a commit to golang/build that referenced this issue Apr 4, 2023
Post-submit builds the same set as other "secondary" longtest builders,
like linux-386-longtest.

Trybot builds are not enabled for release branches due to lack of
capacity.

For golang/go#35678.

Change-Id: I60f15bfe8fe74072f35b20462cd9d13a111a1b31
Reviewed-on: https://go-review.googlesource.com/c/build/+/479837
Reviewed-by: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Michael Pratt <mpratt@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Run-TryBot: Michael Pratt <mpratt@google.com>
@prattmic
Copy link
Member

prattmic commented Apr 4, 2023

This is deployed. Tests are failing because they are attempting to access the GKE proxy: https://build.golang.org/log/3f22deaeecb1b6d29789d0c3dc904a7edf34bf1c.

I don't immediately see why this is happening, or even where this value is coming from. There is no GOPROXY in the logged environment.

@bcmills
Copy link
Contributor Author

bcmills commented Apr 4, 2023

(I don't know why it isn't in the logged environment, though. 🤷‍♂️)

@prattmic
Copy link
Member

prattmic commented Apr 4, 2023

Thanks, I found that but misread some conditions.

(I don't know why it isn't in the logged environment, though. 🤷‍♂️)

I think this is because we are only logging the environment when running make.bash. For tests we add GOPROXY here, only after running make.bash.

@gopherbot
Copy link

Change https://go.dev/cl/482335 mentions this issue: dashborad: remove needsGoProxy

gopherbot pushed a commit to golang/build that referenced this issue Apr 4, 2023
This field was added in CL 166218 but never used.

For golang/go#35678.

Change-Id: I197fd3835136e7649b20a7ca47d4179591cb49a5
Reviewed-on: https://go-review.googlesource.com/c/build/+/482335
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Michael Pratt <mpratt@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/482339 mentions this issue: dashboard,cmd/coordinator: unify and simplify GOPROXY setting behavior

@gopherbot
Copy link

Change https://go.dev/cl/482915 mentions this issue: dashboard: add known issue to darwin-amd64-longtest

gopherbot pushed a commit to golang/build that referenced this issue Apr 6, 2023
The existing behavior for setting GOPROXY is rather hard to follow, and
doesn't work correctly in many cases. For example, longtest on a reverse
builder gets the GKE proxy. Before CL 479837 there were no longtest
builders outside of GCE, so this case was never covered. Fixing this is
the motivation of this CL.

They way configuration works today is:

1. buildstatus.go unconditionally sets GOPROXY to the GKE proxy [1].
2. st.conf.ModuleEnv potentially overrides GOPROXY with a more
   reasonable setting, with a bunch of complex conditions.

Unify and simplify this process by moving it into buildstatus.go, where
their is now a strict ordering of possible GOPROXY values. Notable
changes:

* The GKE proxy is never used outside of GCE.
* There is a consistent default/fallback of proxy.golang.org.

I initially tried to split this into two CLs: one unifying the
implementation and the next changing the behavior, but the old behavior
is so mind-boggling that the first CL doesn't really make much sense.

The annoying part of this CL is that tests move from dashboard to
cmd/coordinator, requiring us to export additional fields so
cmd/coordinator tests can configure the builders.

The test cases themselves are unchanged except for the addition of a
non-GCE longtest builder case.

[1] Except in runSubrepoTests, which avoids doing so for reverse
builders. This was a workaround for private proxy builders in CL 275412,
but wasn't extended to other callers because only subrepo tests were
seeing a regression. More strangeness.

For golang/go#35678.

Change-Id: I6090c8c5e91ce6be9bfc07c16f36ed339c9d27ae
Reviewed-on: https://go-review.googlesource.com/c/build/+/482339
Reviewed-by: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Carlos Amedee <carlos@golang.org>
Run-TryBot: Michael Pratt <mpratt@google.com>
gopherbot pushed a commit to golang/build that referenced this issue Apr 6, 2023
For golang/go#35678.

Change-Id: I1d408c6547e689702311d73b5bf56adf4d1cd150
Reviewed-on: https://go-review.googlesource.com/c/build/+/482915
Run-TryBot: Michael Pratt <mpratt@google.com>
Reviewed-by: Carlos Amedee <carlos@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
@prattmic
Copy link
Member

prattmic commented Apr 6, 2023

The latest failures are in crypto/x509:

https://build.golang.org/log/8815b334c89791bc3c30410757d4c1020e927e96

--- FAIL: TestPlatformVerifier (2.85s)
    --- FAIL: TestPlatformVerifier/revoked_leaf (0.19s)
        root_darwin_test.go:116: unexpected verification error: got "x509: certificate has expired or is not yet valid: “revoked.badssl.com” certificate is expired", want "x509: “revoked.badssl.com” certificate is revoked"
    --- FAIL: TestPlatformVerifier/leaf_missing_SCTs (0.74s)
        root_darwin_test.go:118: unexpected verification success: want "x509: “no-sct.badssl.com” certificate is not standards compliant"
FAIL
FAIL	crypto/x509	4.436s

@rolandshoemaker any idea what is going on here? Do the builders need some certificate verification system dependency installed/enabled?

@bcmills
Copy link
Contributor Author

bcmills commented Apr 6, 2023

FWIW, that's #57428.

@prattmic
Copy link
Member

prattmic commented Apr 6, 2023

Ah, nice! So that test needs a skip for now it seems.

@gopherbot
Copy link

Change https://go.dev/cl/482165 mentions this issue: crypto/x509: skip broken darwin root tests

gopherbot pushed a commit that referenced this issue Apr 10, 2023
For #57428.
For #35678.

Change-Id: I806c16d3ff3815b8681916753338356c444970d2
Reviewed-on: https://go-review.googlesource.com/c/go/+/482165
Reviewed-by: Bryan Mills <bcmills@google.com>
Auto-Submit: Michael Pratt <mpratt@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Run-TryBot: Michael Pratt <mpratt@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/484216 mentions this issue: cmd/go: skip TestScript/list_goroot_symlink on darwin

gopherbot pushed a commit that referenced this issue Apr 13, 2023
The list_goroot_symlink test relies on fsys.Walk (and ultimately
syscall.Lstat) conforming to POSIX pathname resolution semantics.
POSIX requires that symlinks ending in a slash be fully resolved,
but it appears that lstat in current darwin kernels does not fully
resolve the last pathname component when it is a symlink to a symlink.

For #59586.
For #35678.

Change-Id: I37526f012ba94fa1796b33109a41c3226c967d3e
Reviewed-on: https://go-review.googlesource.com/c/go/+/484216
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
TryBot-Bypass: Bryan Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/484295 mentions this issue: cmd/internal/moddeps: preserve PWD more carefully in commands

gopherbot pushed a commit that referenced this issue Apr 13, 2023
On macOS, TMPDIR is typically a symlink, and the GOROOT for the
buildlet is in TMPDIR as well. PWD must be preserved in order for
os.Getwd (and functions based on it) to report paths that remain
relative to GOROOT, and paths relative to GOROOT are necessary in
order for filepath.Rel to report subdirectories as subdirectories
(rather than paths with long "../../…" prefixes).

Fortunately, the (*Cmd).Environ method added for #50599 makes
preserving PWD somewhat easier.

This fixes 'go test cmd/internal/moddeps' on the new
darwin-amd64-longtest builder.

For #35678.

Change-Id: Ibaa458bc9a94b44ba455519bb8da445af07fe0d2
Reviewed-on: https://go-review.googlesource.com/c/go/+/484295
Auto-Submit: Bryan Mills <bcmills@google.com>
Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Michael Pratt <mpratt@google.com>
@gopherbot
Copy link

Change https://go.dev/cl/484476 mentions this issue: dashboard: remove darwin-amd64-longtest known issue

@cagedmantis
Copy link
Contributor

Just mentioning that this post-submit builder is failing in the 1.19 and 1.20 release branches.

@gopherbot
Copy link

Change https://go.dev/cl/492055 mentions this issue: dashboard: make darwin longtest builder like others as of Go 1.21

gopherbot pushed a commit to golang/build that referenced this issue May 3, 2023
The darwin longtest builder was added during the Go 1.21 dev cycle and
needed some development work before it would pass all tests. Disable it
on older release branches that don't have said work, and make it run on
Go 1.21 and newer release branches as a TryBot like all other longtest
builders.

This makes it consistent with other -longtest builders and will help
avoid entering broken state when backporting to future release branches.

Also remove redundant version check from windows longtest tryBot policy
functions since the version check is already there in buildsRepo policy.

For golang/go#35678.
For golang/go#37827.

Change-Id: I09c479f1be2b3635c385b504b0c86b50e65e696b
Reviewed-on: https://go-review.googlesource.com/c/build/+/492055
TryBot-Result: Gopher Robot <gobot@golang.org>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Michael Pratt <mpratt@google.com>
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/503758 mentions this issue: internal/releasetargets: set LongTestBuilder for darwin-amd64

gopherbot pushed a commit to golang/build that referenced this issue Jun 15, 2023
A longtest builder for the darwin/amd64 port was added in the Go 1.21
development cycle. Start using it as for non-advisory release testing.

For golang/go#40561.
For golang/go#35678.
For golang/go#29252.

Change-Id: Ic65e84e5e10bcb786cb28c36c1d1b137e2f6202e
Reviewed-on: https://go-review.googlesource.com/c/build/+/503758
Run-TryBot: Dmitri Shuralyov <dmitshur@golang.org>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Dmitri Shuralyov <dmitshur@google.com>
Reviewed-by: Heschi Kreinick <heschi@google.com>
Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsFix The path to resolution is known, but the work has not been done. new-builder OS-Darwin
Projects
Archived in project
Development

No branches or pull requests

6 participants