Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/build/cmd/release: 2 targets are not passing tests successfully on master branch #39386

Closed
dmitshur opened this issue Jun 3, 2020 · 5 comments
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Jun 3, 2020

The release command is used by releasebot to build targets. In the first stage (-mode=prepare), it runs tests. In second stage (-mode=release), it skips tests.

release supports 12 normal targets (not counting new optional longtest-only targets; described in #29252 but can be ignored in this issue), see here.

These targets are passing tests (they run successfully without -skip_tests flag given to release) on a recent master commit (e.g., 09e791f):

  • src
  • linux-armv6l
  • linux-ppc64le
  • linux-s390x

These are not passing:

Each one can be reproduced if your account has permissions needed to run releasebot (they are documented at https://github.com/golang/build/tree/master/cmd/releasebot#permissions) with:

$ release -target=<target_name> -version=go1.15beta1 -watch -rev=09e791feb1dfbfc05578a17cd9c035ef82af4033

(The release command creates a tarball and places it in a temporary staging directory; it doesn't publish anything, so it is safe to run as shown above.)

See #39385 as an example of a specific test that is failing.

This is the high-level tracking issue for the entire problem. Next, I'll be updating it and splitting into smaller, distinct failures that can be investigated independently.

This issue requires investigation. The problem is either in how the release command sets up the environment and executes commands on the target builder, or in the builder definition, or in the test, or some combination. Depending on what we learn, we may end up deciding this is okay to resolve after beta1 (but before the final release) and skip these test failures in order to make the Go 1.15 beta 1 available for testing and finding unknown issues sooner.

/cc @andybons @toothrot @cagedmantis @golang/osp-team

@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Jun 3, 2020
@dmitshur dmitshur added this to the Go1.15 milestone Jun 3, 2020
@dmitshur dmitshur self-assigned this Jun 5, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Jun 9, 2020

Both known issues #39385 and #39391 have been resolved by now, so I'm testing this again.

I'm using Go commit cacac8b (latest on master branch as of this posting) for this test.

By now, these targets have passed (to be more precise, got as far as failing the API check test, without other failures) at least once:

  • src
  • linux-armv6l
  • linux-ppc64le
  • linux-s390x
  • darwin-amd64
  • freebsd-386
  • freebsd-amd64
  • linux-386
  • linux-amd64
  • linux-arm64

However, windows-386 and windows-amd64 have failed:

windows-386

ok  	cmd/go	111.031s
--- FAIL: TestDWARF (1.28s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	25.084s

windows-amd64

go test proxy running at GOPROXY=http://127.0.0.1:50986/mod
go proxy: no archive rsc.io v1.5.2: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.1.0: file does not exist
go proxy: no archive rsc.io v1.0.0: file does not exist
go proxy: no archive rsc.io v1.5.1: file does not exist
go proxy: no archive example.com/newcycle v1.0.0: file does not exist
go proxy: no archive test.go v1.0.0: file does not exist
go proxy: no archive rsc.io v1.5.2: file does not exist
--- FAIL: TestScript (0.01s)
    --- FAIL: TestScript/test_race_install_cgo (1.21s)
        script_test.go:210: 
            # Tests Issue #10500 (1.209s)
            > [!race] skip
            > [!darwin] ! stale cmd/cgo  # The darwin builders are spuriously stale; see #33598.
            FAIL: testdata\script\test_race_install_cgo.txt:5: cmd/cgo is unexpectedly stale
            
            
FAIL
FAIL	cmd/go	111.443s
--- FAIL: TestDWARF (1.62s)
    dwarf_test.go:39: cmd/link is stale - run go install cmd/link
FAIL
FAIL	cmd/link	19.749s

This may be related to the change in CL 236819. /cc @bcmills

I'll try again a few more times to confirm if it's a flaky or reproducible failure. I'll also try the windows targets with a commit before CL 236819 to see what effect it has. Then update this issue.

@bcmills
Copy link
Contributor

bcmills commented Jun 9, 2020

That one is strange because those two tests are both passing on the regular builders. I'll pop on a gomote and see if they're easy to reproduce with GOROOT_FINAL.

@dmitshur dmitshur changed the title x/build/cmd/release: 8 targets are not passing tests successfully on master branch x/build/cmd/release: 2 targets are not passing tests successfully on master branch Jun 9, 2020
@dmitshur
Copy link
Contributor Author

dmitshur commented Jun 9, 2020

I've tried both windows-amd64 and windows-386 again, and both failed in the same way, so this doesn't seem to be a flake. Filed #39478 as a dedicated tracking issue for it. Thank you again for looking into this @bcmills.

@dmitshur
Copy link
Contributor Author

Issue #39478 has been resolved via CL 237359; a re-test of all targets with it is now ongoing.

@dmitshur
Copy link
Contributor Author

Testing has shown no new test failures. All 12 targets are passing tests when targeting commit 2a4e71c (CL 237359). That means this issue is resolved. Huge thanks to @bcmills for helping make this happen!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Development

No branches or pull requests

3 participants