Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/cmd/coordinator: main Go tests don't pass when CGO_CFLAGS=-mmacosx-version-min=10.13 set in environment #51005

Open
dmitshur opened this issue Feb 3, 2022 · 1 comment
Open

x/build/cmd/coordinator: main Go tests don't pass when CGO_CFLAGS=-mmacosx-version-min=10.13 set in environment #51005

dmitshur opened this issue Feb 3, 2022 · 1 comment
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@dmitshur
Copy link
Contributor

dmitshur commented Feb 3, 2022 •
edited

CL 382055 set CGO_CFLAGS=-mmacosx-version-min=10.13 in the environment of the darwin-amd64-10_15 builder. With this change, all golang.org/x module tests continued to pass (e.g., x/arch build log), but the main Go tests started to fail with the following output:

darwin-amd64-10_15 at 896df422a7cecbace10f5877beeeb1476b6061ae

go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE archive/tar: stale dependency: runtime/cgo
	STALE crypto/tls: stale dependency: runtime/cgo
	STALE crypto/x509: stale dependency: runtime/cgo
[...]
	STALE cmd/vendor/github.com/google/pprof/internal/symbolz: stale dependency: runtime/cgo
	STALE cmd/vendor/github.com/google/pprof/internal/transport: stale dependency: runtime/cgo
	STALE cmd/vendor/golang.org/x/mod/sumdb: stale dependency: runtime/cgo

(Source: https://build.golang.org/log/4474c782b89f18405361027bebcab80e09129190.)

Another test run, with more information captured, suggests the problem happens in the "run_tests_multi" step when the sharded tests start running:

  builder: darwin-amd64-10_15
      rev: 896df422a7cecbace10f5877beeeb1476b6061ae
 buildlet: http://macstadium_host03a reverse peer macstadium_host03a/207.254.3.10:56081 for host type host-darwin-10_15
  started: 2022-02-03 21:55:43.764155923 +0000 UTC m=+2657.016357431
    ended: 2022-02-03 22:03:37.386828895 +0000 UTC m=+3130.639030413
  success: false

Events:
  2022-02-03T21:55:43Z checking_for_snapshot 
  2022-02-03T21:55:43Z finish_checking_for_snapshot after 25.3ms
  2022-02-03T21:55:43Z get_buildlet 
  2022-02-03T22:03:26Z finish_get_buildlet after 7m42.1s
  2022-02-03T22:03:26Z using_buildlet macstadium_host03a
  2022-02-03T22:03:26Z write_snapshot_tar 
  2022-02-03T22:03:34Z finish_write_snapshot_tar after 7.6s
  2022-02-03T22:03:34Z make_and_test 
  2022-02-03T22:03:34Z discovering_tests 
  2022-02-03T22:03:35Z get_test_stats 
  2022-02-03T22:03:35Z finish_get_test_stats after 0s
  2022-02-03T22:03:35Z starting_tests 270 tests
  2022-02-03T22:03:35Z run_tests_multi macstadium_host03a: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2022-02-03T22:03:37Z finish_run_tests_multi after 1.31s; macstadium_host03a: [go_test:archive/tar go_test:archive/zip go_test:bufio]
  2022-02-03T22:03:37Z run_tests_multi macstadium_host03a: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2022-02-03T22:03:37Z finish_make_and_test after 2.94s
  2022-02-03T22:03:37Z done with test failures
  2022-02-03T22:03:37Z finish_run_tests_multi after 209.7ms; err=buildlet: Client closed; macstadium_host03a: [go_test:bytes go_test:compress/bzip2 go_test:compress/flate]
  2022-02-03T22:03:37Z main_buildlet_broken macstadium_host03a

Build log:
darwin-amd64-10_15 at 896df422a7cecbace10f5877beeeb1476b6061ae

go tool dist: unexpected stale targets reported by go list -gcflags="" -ldflags="" for [std cmd] (consider rerunning with GOMAXPROCS=1 GODEBUG=gocachehash=1):
	STALE archive/tar: stale dependency: runtime/cgo
	STALE crypto/tls: stale dependency: runtime/cgo
	STALE crypto/x509: stale dependency: runtime/cgo
[...]
	STALE cmd/vendor/github.com/google/pprof/internal/symbolz: stale dependency: runtime/cgo
	STALE cmd/vendor/github.com/google/pprof/internal/transport: stale dependency: runtime/cgo
	STALE cmd/vendor/golang.org/x/mod/sumdb: stale dependency: runtime/cgo



Error: tests failed: dist test failed: go_test:archive/tar: exit status 1

CC @golang/release, @bcmills.
@dmitshur dmitshur added Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 3, 2022
@dmitshur dmitshur added this to the Unreleased milestone Feb 3, 2022
@dmitshur dmitshur mentioned this issue Feb 7, 2022
Open
@dmitshur
Copy link
Contributor Author

dmitshur commented Mar 28, 2022 •
edited

Writing down that when this was discussed previously, there was a hypothesis that this might've a false positive caused by a race involving the old snapshot of a built Go toolchain (built before the coordinator was redeployed with the CGO_CFLAGS env var in the builder configuration) being reused in the new build, since the snapshot is keyed only on the builder name and Go commit hash.

If that turns out to be it, we can consider improving snapshot key to include some a hash of the builder configuration or coordinator version.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Builders x/build issues (builders, bots, dashboards) NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
1 participant
@dmitshur