Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/build/internal/relui: TestRelease failures #55178

Closed
gopherbot opened this issue Sep 20, 2022 · 8 comments
Closed

x/build/internal/relui: TestRelease failures #55178

gopherbot opened this issue Sep 20, 2022 · 8 comments
Assignees
@dmitshur
Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Unreleased

Comments

@gopherbot
Copy link 

#!watchflakes
post <- pkg == "golang.org/x/build/internal/relui" && test == "TestRelease"

Bug automatically created to track these flakes.

— watchflakes
@gopherbot gopherbot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 20, 2022
@gopherbot
Copy link
Author

gopherbot commented Sep 20, 2022 •
edited

Found new matching flaky dashboard failures for:

#!watchflakes
post <- pkg == "golang.org/x/build/internal/relui" && test == "TestRelease"
2022-07-16 04:49 darwin-arm64-12 build@77d1e5c1 go@de8101d2 golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/linux-arm-aws/1: 5 files, 2 dirs (1.525208ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/windows-amd64-2008/2: 5 files, 2 dirs (1.761625ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/darwin-amd64-12_0/0: 5 files, 2 dirs (3.132375ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/windows-arm64-11/3: 5 files, 2 dirs (1.324583ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/windows-386-2008/4: 5 files, 2 dirs (1.751916ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/windows-amd64-2008/2/go1.4: 1 files, 1 dirs (623.833µs)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/linux-386-buster/6: 5 files, 2 dirs (7.578875ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/freebsd-amd64-12_3/8: 5 files, 2 dirs (3.984875ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/linux-ppc64le-buildlet/5: 5 files, 2 dirs (10.310542ms)
2022/07/18 14:14:18 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta4218877240/001/linux-arm64-aws/7: 5 files, 2 dirs (22.407625ms)
...
--- FAIL: TestRelease (2.05s)
    --- FAIL: TestRelease/beta (1.15s)
        buildrelease_test.go:747: task linux-arm64: Build binary archive: started
        buildrelease_test.go:747: task darwin-amd64: Run short tests: started
        buildrelease_test.go:747: task Tag version: started
        buildrelease_test.go:747: task Pick milestones: started
        buildrelease_test.go:747: task Upload artifacts to CDN: started
        buildrelease_test.go:747: task windows-amd64: Build MSI: started
        buildrelease_test.go:747: task linux-armv6l: Run short tests: started
        buildrelease_test.go:747: task windows-arm64: Convert to .zip: started
...
I'm a test! :D
2022/07/18 14:14:21 extracted tarball into /tmp/buildlet/tmp/TestSecurityfailure1361924808/001/freebsd-amd64-12_3/29: 10 files, 7 dirs (1.7015ms)
2022/07/18 14:14:21 extracted tarball into /tmp/buildlet/tmp/TestSecurityfailure1361924808/001/freebsd-amd64-12_3/29/go1.4: 1 files, 1 dirs (630.458µs)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/18 14:14:21 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-20 20:17 darwin-amd64-12_0 build@d78880f6 go@bb1749ba golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/darwin-amd64-12_0/1: 5 files, 2 dirs (4.919641ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/freebsd-386-12_3/0: 5 files, 2 dirs (5.230009ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/windows-arm64-11/5: 5 files, 2 dirs (5.067334ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/linux-s390x-crosscompile/6: 5 files, 2 dirs (4.698784ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/linux-amd64-buster/4: 5 files, 2 dirs (4.201443ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/freebsd-amd64-12_3/2: 5 files, 2 dirs (12.597262ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/linux-386-buster/10: 5 files, 2 dirs (15.703889ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/linux-arm-aws/12: 5 files, 2 dirs (5.77027ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/linux-arm64-aws/11: 5 files, 2 dirs (22.222691ms)
2022/07/20 13:19:28 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3575177693/001/freebsd-386-12_3/0/go1.4: 1 files, 1 dirs (8.822532ms)
...
--- FAIL: TestRelease (7.60s)
    --- FAIL: TestRelease/beta (3.95s)
        buildrelease_test.go:747: task Push issues: started
        buildrelease_test.go:747: task windows-arm64: Convert to .zip: started
        buildrelease_test.go:747: task Stage artifacts for signing: started
        buildrelease_test.go:747: task Upload artifacts to CDN: started
        buildrelease_test.go:747: task Wait for DL CL: started
        buildrelease_test.go:747: task Read starting branch head: started
        buildrelease_test.go:747: task linux-ppc64le: Build binary archive: started
        buildrelease_test.go:747: task Mail DL CL: started
...
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/20 13:19:41 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-25 16:43 darwin-amd64-10_15 build@a3d9d89c go@12e00f6c golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/darwin-amd64-12_0/7: 5 files, 2 dirs (2.431691ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/windows-arm64-11/8: 5 files, 2 dirs (2.373685ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/windows-amd64-2008/9: 5 files, 2 dirs (2.546958ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/darwin-arm64-12/4: 5 files, 2 dirs (4.18668ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/freebsd-amd64-12_3/0: 5 files, 2 dirs (7.227808ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/freebsd-386-12_3/6: 5 files, 2 dirs (14.750375ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/linux-s390x-crosscompile/10: 5 files, 2 dirs (16.543614ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/linux-amd64-buster/1: 5 files, 2 dirs (24.27047ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/linux-ppc64le-buildlet/11: 5 files, 2 dirs (18.276941ms)
2022/07/25 12:49:27 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2463353378/001/linux-arm64-aws/12: 5 files, 2 dirs (10.818726ms)
...
--- FAIL: TestRelease (4.98s)
    --- FAIL: TestRelease/rc (2.02s)
        buildrelease_test.go:747: task Stage artifacts for signing: started
        buildrelease_test.go:747: task darwin-arm64: Run short tests: started
        buildrelease_test.go:747: task darwin-amd64: Build binary archive: started
        buildrelease_test.go:747: task freebsd-amd64: Build binary archive: started
        buildrelease_test.go:747: task windows-386: Run short tests: started
        buildrelease_test.go:747: task linux-386: Build binary archive: started
        buildrelease_test.go:747: task linux-ppc64le: Build binary archive: started
        buildrelease_test.go:747: task Read current branch head: started
...
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/25 12:49:36 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-27 15:11 darwin-amd64-12_0 build@fa54b016 go@be7c6814 golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/linux-ppc64le-buildlet/50: 7 files, 2 dirs (24.999225ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/linux-386-stretch/5: 7 files, 2 dirs (46.027236ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/windows-amd64-race/12: 7 files, 2 dirs (50.195566ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/linux-amd64-nocgo/23: 7 files, 2 dirs (85.096204ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/openbsd-amd64-68/24: 7 files, 2 dirs (84.110876ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/openbsd-386-68/49: 7 files, 2 dirs (84.950381ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/linux-amd64-goamd64v3/2: 7 files, 2 dirs (107.155664ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/linux-amd64-boringcrypto/8: 7 files, 2 dirs (110.814006ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/windows-amd64-2012/32: 7 files, 2 dirs (93.434243ms)
2022/07/27 11:14:31 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1444955480/001/windows-arm64-11/34: 7 files, 2 dirs (95.124301ms)
...
--- FAIL: TestRelease (12.31s)
    --- FAIL: TestRelease/beta (6.84s)
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-bullseye: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-boringcrypto: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-race: started
        buildrelease_test.go:796: task darwin-amd64: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-noopt: started
        buildrelease_test.go:796: task Wait for signed artifacts: started
        buildrelease_test.go:796: task Upload artifacts to CDN: started
        buildrelease_test.go:796: task Wait for signing and tests: started
...
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/27 11:15:08 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-28 17:25 darwin-amd64-10_15 build@2c833c66 go@d9242f7a golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/windows-amd64-2008-newcc/6: 7 files, 2 dirs (12.451048ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/windows-amd64-2012-newcc/3: 7 files, 2 dirs (16.536359ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/darwin-arm64-12/15: 7 files, 2 dirs (6.755745ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/freebsd-amd64-race/2: 7 files, 2 dirs (7.293993ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/windows-386-2012/16: 7 files, 2 dirs (18.966268ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/linux-ppc64le-buildlet/12: 7 files, 2 dirs (26.008344ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/android-amd64-emu/7: 7 files, 2 dirs (19.351616ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/windows-amd64-longtest-newcc/21: 7 files, 2 dirs (17.024885ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/linux-s390x-crosscompile/22: 7 files, 2 dirs (17.248265ms)
2022/07/28 13:27:23 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta4094598879/001/linux-amd64-ssacheck/18: 7 files, 2 dirs (28.24473ms)
...
--- FAIL: TestRelease (12.47s)
    --- FAIL: TestRelease/rc (5.72s)
        buildrelease_test.go:796: task Get next version: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-boringcrypto: started
        buildrelease_test.go:796: task windows-amd64: Run long tests: started
        buildrelease_test.go:796: task linux-s390x: Build binary archive: started
        buildrelease_test.go:796: task freebsd-386: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot freebsd-amd64-13_0: started
        buildrelease_test.go:796: task windows-arm64: Run short tests: started
        buildrelease_test.go:796: task Wait for Release Coordinator Approval: started
...
I'm a test! :D
2022/07/28 13:27:55 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestAdvisoryTrybotFail723636703/001/darwin-amd64-12_0/71: 11 files, 7 dirs (6.765826ms)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/28 13:27:56 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-29 01:18 darwin-amd64-12_0 build@540cfda9 go@be7c6814 golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-amd64-race/12: 7 files, 2 dirs (22.854171ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/netbsd-386-9_0/15: 7 files, 2 dirs (24.987554ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-amd64-staticlockranking/8: 7 files, 2 dirs (26.936362ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-s390x-crosscompile/16: 7 files, 2 dirs (4.175986ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-386/14: 7 files, 2 dirs (43.169291ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/windows-amd64-2012-newcc/24: 7 files, 2 dirs (42.500034ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/windows-amd64-2008/0: 7 files, 2 dirs (56.440789ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-386-buster/7: 7 files, 2 dirs (58.273899ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-ppc64le-buildlet/10: 7 files, 2 dirs (60.584157ms)
2022/07/28 18:20:43 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1281478932/001/linux-386-clang/17: 7 files, 2 dirs (67.455787ms)
...
--- FAIL: TestRelease (15.51s)
    --- FAIL: TestRelease/beta (7.66s)
        buildrelease_test.go:796: task Run advisory TryBot netbsd-amd64-9_0: started
        buildrelease_test.go:796: task Upload artifacts to CDN: started
        buildrelease_test.go:796: task windows-arm64: Convert to .zip: started
        buildrelease_test.go:796: task windows-amd64: Convert to .zip: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-race: started
        buildrelease_test.go:796: task linux-armv6l: Run short tests: started
        buildrelease_test.go:796: task Run advisory TryBot js-wasm: started
        buildrelease_test.go:796: task linux-ppc64le: Build binary archive: started
...
2022/07/28 18:21:12 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2001265520/001/darwin-arm64-12/70: 11 files, 7 dirs (5.685516ms)
2022/07/28 18:21:12 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2001265520/001/darwin-amd64-12_0/71: 11 files, 7 dirs (3.794477ms)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/28 18:21:13 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-07-29 15:52 darwin-amd64-12_0 build@ebbd3258 go@9240558e golang.org/x/build/internal/relui.TestRelease (log) 
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/linux-amd64-noopt/5: 7 files, 2 dirs (4.158077ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/netbsd-amd64-9_0/6: 7 files, 2 dirs (5.99671ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/android-386-emu/8: 7 files, 2 dirs (6.229145ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/linux-amd64-buster/7: 7 files, 2 dirs (5.863889ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/openbsd-386-68/3: 7 files, 2 dirs (7.147015ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/linux-386-buster/1: 7 files, 2 dirs (9.885988ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/linux-amd64-staticlockranking/10: 7 files, 2 dirs (9.38675ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/windows-386-2008-newcc/9: 7 files, 2 dirs (7.481426ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/windows-amd64-longtest-newcc/0: 7 files, 2 dirs (8.12766ms)
2022/07/29 08:54:05 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2210979362/001/openbsd-386-68/3/go1.4: 1 files, 1 dirs (1.720119ms)
...
--- FAIL: TestRelease (10.90s)
    --- FAIL: TestRelease/rc (6.05s)
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-race: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64: started
        buildrelease_test.go:796: task Get next version: started
        buildrelease_test.go:796: task Mail DL CL: started
        buildrelease_test.go:796: task Run advisory TryBot windows-386-2012: started
        buildrelease_test.go:796: task Run advisory TryBot windows-386-2012-newcc: started
        buildrelease_test.go:796: task Wait for Release Coordinator Approval: started
        buildrelease_test.go:796: task darwin-arm64: Build binary archive: started
...
2022/07/29 08:54:29 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2120627540/001/windows-amd64-2008/70: 11 files, 7 dirs (6.497302ms)
2022/07/29 08:54:29 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2120627540/001/windows-amd64-2008/71: 11 files, 7 dirs (7.270646ms)
2022/07/29 08:54:29 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2120627540/001/windows-amd64-2008/70/go1.4: 1 files, 1 dirs (539.864µs)
2022/07/29 08:54:29 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail2120627540/001/windows-amd64-2008/69/go1.4: 1 files, 1 dirs (582.802µs)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/07/29 08:54:30 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-01 15:11 darwin-amd64-10_15 build@0a9b2ecb go@15da892a golang.org/x/build/internal/relui.TestRelease (log) 
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-amd64-2008/10: 7 files, 2 dirs (5.688901ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-amd64-race/1: 7 files, 2 dirs (6.337898ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/freebsd-amd64-13_0/0: 7 files, 2 dirs (4.897706ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/freebsd-386-12_3/9: 7 files, 2 dirs (9.91498ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-amd64-2008/10/go1.4: 1 files, 1 dirs (1.865246ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/android-386-emu/2: 7 files, 2 dirs (10.034339ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/freebsd-amd64-race/6: 7 files, 2 dirs (9.459359ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-amd64-2012/14: 7 files, 2 dirs (7.728825ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-386-2012-newcc/15: 7 files, 2 dirs (6.729965ms)
2022/08/01 11:43:20 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta2959117402/001/windows-amd64-2016-newcc/13: 7 files, 2 dirs (16.244938ms)
...
--- FAIL: TestRelease (15.72s)
    --- FAIL: TestRelease/beta (7.90s)
        buildrelease_test.go:796: task Publish to website: started
        buildrelease_test.go:796: task linux-armv6l: Build binary archive: started
        buildrelease_test.go:796: task Approve any TryBot failures: started
        buildrelease_test.go:796: task linux-arm64: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot linux-386-softfloat: started
        buildrelease_test.go:796: task Run advisory TryBot windows-amd64-longtest-newcc: started
        buildrelease_test.go:796: task Wait for signed artifacts: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-racecompile: started
...
2022/08/01 11:43:52 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestAdvisoryTrybotFail3952310703/001/darwin-amd64-12_0/71: 11 files, 7 dirs (6.498483ms)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/08/01 11:43:52 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-01 15:11 linux-amd64-race build@0a9b2ecb go@be59153d golang.org/x/build/internal/relui.TestRelease (log) 
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/windows-arm64-11/2: 7 files, 2 dirs (5.287125ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/windows-386-2008/0: 7 files, 2 dirs (4.829211ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/windows-amd64-2016-newcc/1: 7 files, 2 dirs (3.223892ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/windows-amd64-2012/3: 7 files, 2 dirs (6.730548ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/linux-386-clang/6: 7 files, 2 dirs (5.547735ms)
I'm a test! :D
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/freebsd-386-12_3/5: 7 files, 2 dirs (10.066677ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/windows-amd64-2008/4: 7 files, 2 dirs (9.443794ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/freebsd-386-13_0/9: 7 files, 2 dirs (7.167895ms)
2022/08/01 15:25:39 extracted tarball into /workdir/tmp/TestReleasebeta2999066681/001/linux-amd64-noopt/14: 7 files, 2 dirs (3.317301ms)
...
--- FAIL: TestRelease (1.66s)
    --- FAIL: TestRelease/rc (0.82s)
        buildrelease_test.go:796: task linux-amd64: Run long tests: started
        buildrelease_test.go:796: task Get next version: started
        buildrelease_test.go:796: task Run advisory TryBot android-386-emu: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-bullseye: started
        buildrelease_test.go:796: task windows-arm64: Run short tests: started
        buildrelease_test.go:796: task freebsd-amd64: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot windows-amd64-newcc-race: started
        buildrelease_test.go:796: task linux-armv6l: Run short tests: started
...
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/windows-amd64-2008/64/go1.4: 1 files, 1 dirs (624.596µs)
I'm a test! :D
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/freebsd-386-12_3/69: 10 files, 5 dirs (4.717955ms)
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/windows-arm64-11/70: 10 files, 5 dirs (6.040804ms)
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/windows-arm64-11/71: 10 files, 5 dirs (6.722994ms)
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/freebsd-386-12_3/69/go1.4: 1 files, 1 dirs (793.076µs)
I'm a test! :D
2022/08/01 15:25:43 extracted tarball into /workdir/tmp/TestAdvisoryTrybotFail3487312856/001/windows-arm64-11/70/go1.4: 1 files, 1 dirs (847.687µs)
I'm a test! :D
2022/08/01 15:25:43 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-02 02:32 darwin-amd64-10_15 build@4b12e2b4 go@be59153d golang.org/x/build/internal/relui.TestRelease (log) 
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-amd64-boringcrypto/12: 7 files, 2 dirs (12.31935ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-s390x-crosscompile/7: 7 files, 2 dirs (13.942397ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-amd64-racecompile/3: 7 files, 2 dirs (18.806477ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/android-amd64-emu/4: 7 files, 2 dirs (40.954223ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/windows-amd64-2016-newcc/1: 7 files, 2 dirs (42.348531ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-amd64-race/10: 7 files, 2 dirs (46.213798ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-386-clang/8: 7 files, 2 dirs (47.829827ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/freebsd-amd64-13_0/14: 7 files, 2 dirs (27.953354ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/darwin-amd64-12_0/6: 7 files, 2 dirs (48.749005ms)
2022/08/01 22:33:58 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestReleasebeta972939622/001/linux-386/2: 7 files, 2 dirs (48.676118ms)
...
--- FAIL: TestRelease (12.38s)
    --- FAIL: TestRelease/beta (6.23s)
        buildrelease_test.go:796: task darwin-arm64: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-nocgo: started
        buildrelease_test.go:796: task Pick milestones: started
        buildrelease_test.go:796: task freebsd-amd64: Run short tests: started
        buildrelease_test.go:796: task linux-ppc64le: Build binary archive: started
        buildrelease_test.go:796: task darwin-arm64: Run short tests: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-boringcrypto: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-race: started
...
2022/08/01 22:34:29 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestAdvisoryTrybotFail1539439272/001/linux-amd64-stretch/71: 11 files, 7 dirs (7.628037ms)
2022/08/01 22:34:29 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-10_15/tmp/TestAdvisoryTrybotFail1539439272/001/linux-amd64-stretch/70: 11 files, 7 dirs (9.576797ms)
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
I'm a test! :D
2022/08/01 22:34:30 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-10 22:55 darwin-arm64-11 build@27753f8c go@a1337ef4 golang.org/x/build/internal/relui.TestRelease (log) 
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/freebsd-amd64-race/7: 7 files, 2 dirs (2.389833ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/linux-s390x-crosscompile/6: 7 files, 2 dirs (2.992708ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/windows-arm64-11/5: 7 files, 2 dirs (3.274084ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/windows-amd64-newcc-race/8: 7 files, 2 dirs (3.258ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/linux-amd64-buster/0: 7 files, 2 dirs (5.557792ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/linux-s390x-crosscompile/6/go1.4: 1 files, 1 dirs (331.333µs)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/linux-arm64-aws/4: 7 files, 2 dirs (6.22125ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/windows-amd64-newcc-race/8/go1.4: 1 files, 1 dirs (220.458µs)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/windows-386-2008/13: 7 files, 2 dirs (2.961958ms)
2022/08/11 10:03:26 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta2261804160/001/linux-amd64-buster/0/go1.4: 1 files, 1 dirs (234.416µs)
...
--- FAIL: TestRelease (4.46s)
    --- FAIL: TestRelease/rc (2.07s)
        buildrelease_test.go:796: task Pick milestones: started
        buildrelease_test.go:796: task Run advisory TryBot dragonfly-amd64-622: started
        buildrelease_test.go:796: task linux-arm64: Build binary archive: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-bullseye: started
        buildrelease_test.go:796: task Run advisory TryBot windows-386-2008-newcc: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-goamd64v3: started
        buildrelease_test.go:796: task Run advisory TryBot windows-amd64-2008-newcc: started
        buildrelease_test.go:796: task linux-386: Build binary archive: started
...
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/freebsd-amd64-12_3/68: 11 files, 7 dirs (1.11275ms)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/freebsd-amd64-12_3/68/go1.4: 1 files, 1 dirs (184.042µs)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/windows-amd64-2008/69: 11 files, 7 dirs (2.630333ms)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/windows-amd64-2008/71: 11 files, 7 dirs (1.8525ms)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/windows-amd64-2008/70: 11 files, 7 dirs (3.0285ms)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/windows-amd64-2008/71/go1.4: 1 files, 1 dirs (226.541µs)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/windows-amd64-2008/69/go1.4: 1 files, 1 dirs (392.292µs)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/darwin-amd64-12_0/72: 11 files, 7 dirs (2.263917ms)
2022/08/11 10:03:35 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3282464010/001/darwin-amd64-12_0/72/go1.4: 1 files, 1 dirs (182.125µs)
2022/08/11 10:03:36 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-31 17:29 darwin-amd64-11_0 build@d7b416f6 go@0592ce5f golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/linux-amd64-race/4: 7 files, 2 dirs (5.353655ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/windows-amd64-2008/11: 7 files, 2 dirs (5.208885ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/windows-386-2008-newcc/6: 7 files, 2 dirs (9.243055ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/windows-amd64-2012-newcc/1: 7 files, 2 dirs (3.931007ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/windows-386-2012-newcc/12: 7 files, 2 dirs (8.209788ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/openbsd-386-70/15: 7 files, 2 dirs (5.120533ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/linux-amd64-boringcrypto/8: 7 files, 2 dirs (3.999545ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/linux-amd64/14: 7 files, 2 dirs (7.682753ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/windows-amd64-2008/11/go1.4: 1 files, 1 dirs (2.310187ms)
2022/09/01 17:11:07 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1720397692/001/netbsd-386-9_0/3: 7 files, 2 dirs (18.24338ms)
...
--- FAIL: TestRelease (9.38s)
    --- FAIL: TestRelease/rc (4.74s)
        buildrelease_test.go:796: task linux-amd64: Run long tests: started
        buildrelease_test.go:796: task linux-armv6l: Build binary archive: started
        buildrelease_test.go:796: task Read current branch head: started
        buildrelease_test.go:796: task Wait for signing and tests: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-staticlockranking: started
        buildrelease_test.go:796: task windows-amd64: Build MSI: started
        buildrelease_test.go:796: task Run advisory TryBot windows-amd64-longtest-newcc: started
        buildrelease_test.go:796: task windows-amd64: Build binary archive: started
...
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/linux-amd64-stretch/69: 11 files, 7 dirs (5.015125ms)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/linux-amd64-stretch/69/go1.4: 1 files, 1 dirs (653.152µs)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/linux-amd64-stretch/68/go1.4: 1 files, 1 dirs (669.614µs)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/darwin-amd64-12_0/70: 11 files, 7 dirs (3.788584ms)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/darwin-amd64-12_0/70/go1.4: 1 files, 1 dirs (767.01µs)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/darwin-arm64-12/71: 11 files, 7 dirs (3.665226ms)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/darwin-arm64-12/71/go1.4: 1 files, 1 dirs (870.624µs)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/freebsd-amd64-12_3/72: 11 files, 7 dirs (3.844833ms)
2022/09/01 17:11:25 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail1685427148/001/freebsd-amd64-12_3/72/go1.4: 1 files, 1 dirs (621.051µs)
2022/09/01 17:11:26 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-08-31 17:29 darwin-arm64-12 build@d7b416f6 go@e4b624ea golang.org/x/build/internal/relui.TestRelease (log) 
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/openbsd-amd64-68/12: 7 files, 2 dirs (1.21925ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/linux-amd64-stretch/10: 7 files, 2 dirs (822.792µs)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/freebsd-386-12_3/3: 7 files, 2 dirs (6.022833ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/freebsd-386-13_0/0: 7 files, 2 dirs (5.451708ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/freebsd-386-13_0/0/go1.4: 1 files, 1 dirs (985.625µs)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/linux-amd64-stretch/10/go1.4: 1 files, 1 dirs (1.119125ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/linux-amd64-goamd64v3/7: 7 files, 2 dirs (9.845625ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/linux-arm-aws/2: 7 files, 2 dirs (10.760167ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/freebsd-386-12_3/3/go1.4: 1 files, 1 dirs (1.642459ms)
2022/08/31 18:28:56 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta71017478/001/darwin-amd64-12_0/5: 7 files, 2 dirs (10.811667ms)
...
--- FAIL: TestRelease (18.88s)
    --- FAIL: TestRelease/beta (9.83s)
        buildrelease_test.go:796: task linux-arm64: Run short tests: started
        buildrelease_test.go:796: task Run advisory TryBot android-amd64-emu: started
        buildrelease_test.go:796: task Tag version: started
        buildrelease_test.go:796: task Run advisory TryBot linux-386-softfloat: started
        buildrelease_test.go:796: task Run advisory TryBot openbsd-386-68: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-boringcrypto: started
        buildrelease_test.go:796: task linux-386: Run long tests: started
        buildrelease_test.go:796: task Run advisory TryBot linux-arm64-boringcrypto: started
...
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/linux-amd64-stretch/67/go1.4: 1 files, 1 dirs (408.541µs)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/linux-amd64-stretch/68/go1.4: 1 files, 1 dirs (533.125µs)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/windows-amd64-2008/69: 11 files, 7 dirs (2.422ms)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/windows-amd64-2008/71: 11 files, 7 dirs (2.835833ms)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/windows-amd64-2008/70: 11 files, 7 dirs (3.181333ms)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/windows-amd64-2008/69/go1.4: 1 files, 1 dirs (724.916µs)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/windows-amd64-2008/70/go1.4: 1 files, 1 dirs (390.375µs)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/darwin-arm64-12/72: 11 files, 7 dirs (1.188042ms)
2022/08/31 18:29:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2177799590/001/darwin-arm64-12/72/go1.4: 1 files, 1 dirs (170.417µs)
2022/08/31 18:29:37 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-09-02 22:19 darwin-amd64-11_0 build@563d5b8b go@cdb270cf golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/linux-arm-aws/1: 7 files, 2 dirs (3.554252ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/windows-386-2012-newcc/5: 7 files, 2 dirs (6.139393ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/freebsd-386-12_3/0: 7 files, 2 dirs (7.364054ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/linux-arm64-aws/4: 7 files, 2 dirs (6.774941ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/linux-amd64-sid/2: 7 files, 2 dirs (8.308736ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/windows-amd64-2008-newcc/8: 7 files, 2 dirs (2.456688ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/openbsd-amd64-68/6: 7 files, 2 dirs (6.989151ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/windows-amd64-longtest-newcc/3: 7 files, 2 dirs (5.091952ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/linux-amd64-ssacheck/9: 7 files, 2 dirs (10.779248ms)
2022/09/07 09:56:26 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta2183732845/001/windows-amd64-newcc-race/11: 7 files, 2 dirs (6.014062ms)
...
--- FAIL: TestRelease (8.63s)
    --- FAIL: TestRelease/rc (4.08s)
        buildrelease_test.go:796: task linux-amd64: Build binary archive: started
        buildrelease_test.go:796: task windows-amd64: Build MSI: started
        buildrelease_test.go:796: task freebsd-386: Run short tests: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-racecompile: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-noopt: started
        buildrelease_test.go:796: task Run advisory TryBot openbsd-386-70: started
        buildrelease_test.go:796: task Run advisory TryBot linux-amd64-clang: started
        buildrelease_test.go:796: task windows-arm64: Run short tests: started
...
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/windows-amd64-2008/68: 11 files, 7 dirs (6.216165ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/windows-amd64-2008/69: 11 files, 7 dirs (10.326263ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/windows-amd64-2008/69/go1.4: 1 files, 1 dirs (1.108686ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/windows-amd64-2008/70: 11 files, 7 dirs (9.627656ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/windows-amd64-2008/70/go1.4: 1 files, 1 dirs (873.598µs)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/darwin-amd64-12_0/71: 11 files, 7 dirs (4.774261ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/darwin-amd64-12_0/71/go1.4: 1 files, 1 dirs (534.286µs)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/darwin-arm64-12/72: 11 files, 7 dirs (4.433679ms)
2022/09/07 09:56:43 extracted tarball into /Users/gopher/workdir/tmp/TestAdvisoryTrybotFail985434101/001/darwin-arm64-12/72/go1.4: 1 files, 1 dirs (517.781µs)
2022/09/07 09:56:44 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7
2022-09-02 22:19 darwin-arm64-12 build@563d5b8b go@ef69718d golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-amd64-noopt/6: 7 files, 2 dirs (3.285584ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-ppc64le-buildlet/4: 7 files, 2 dirs (5.215375ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-amd64-sid/1: 7 files, 2 dirs (6.103583ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-386-clang/3: 7 files, 2 dirs (7.852334ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-amd64-fedora/5: 7 files, 2 dirs (9.685916ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/windows-386-2008-newcc/7: 7 files, 2 dirs (5.552292ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/windows-amd64-longtest-newcc/18: 7 files, 2 dirs (962.792µs)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/windows-386-2008/10: 7 files, 2 dirs (984µs)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/windows-amd64-2016-newcc/2: 7 files, 2 dirs (6.481542ms)
2022/09/06 14:22:23 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta1388479594/001/linux-arm64-boringcrypto/14: 7 files, 2 dirs (6.221542ms)
...
--- FAIL: TestRelease (6.03s)
    --- FAIL: TestRelease/beta (2.99s)
        buildrelease_test.go:796: task linux-amd64: Run long tests: started
        buildrelease_test.go:796: task darwin-arm64: Build binary archive: started
        buildrelease_test.go:796: task linux-s390x: Build binary archive: started
        buildrelease_test.go:796: task linux-armv6l: Run short tests: started
        buildrelease_test.go:796: task windows-arm64: Run short tests: started
        buildrelease_test.go:796: task Push issues: started
        buildrelease_test.go:796: task freebsd-386: Run short tests: started
        buildrelease_test.go:796: task Run advisory TryBot windows-amd64-2012-newcc: started
...
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/windows-amd64-2008/69: 11 files, 7 dirs (11.113042ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/windows-amd64-2008/68: 11 files, 7 dirs (17.314708ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/windows-amd64-2008/68/go1.4: 1 files, 1 dirs (263.875µs)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/windows-amd64-2008/70: 11 files, 7 dirs (32.717ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/windows-amd64-2008/69/go1.4: 1 files, 1 dirs (4.06375ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/linux-amd64-stretch/71: 11 files, 7 dirs (10.506625ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/linux-amd64-stretch/72: 11 files, 7 dirs (9.571916ms)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/linux-amd64-stretch/71/go1.4: 1 files, 1 dirs (302.625µs)
2022/09/06 14:22:36 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail2817099605/001/linux-amd64-stretch/72/go1.4: 1 files, 1 dirs (665.375µs)
2022/09/06 14:22:37 stopWorkflowHandler(_, _, [{id invalid}]) uuid.Parse(invalid): invalid UUID length: 7

— watchflakes

@gopherbot gopherbot added the Builders x/build issues (builders, bots, dashboards) label Sep 20, 2022
@gopherbot gopherbot added this to the Unreleased milestone Sep 20, 2022
@rsc
Copy link
Contributor

rsc commented Sep 20, 2022

This stopped between 8/10 and 8/31, and then started again, and has been silent since 9/2.
Perhaps it is over, or perhaps not.

@gopherbot gopherbot changed the title x/build/internal/relui: flaky TestRelease x/build/internal/relui: TestRelease failures Sep 20, 2022
@gopherbot
Copy link
Author

Found new matching dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/build/internal/relui" && test == "TestRelease"
2022-09-08 09:47 darwin-amd64-10_15 build@b4daf1e8 go@29604312 golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/linux-amd64-sid/5: 7 files, 2 dirs (5.260729ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/linux-arm64-aws/3: 7 files, 2 dirs (5.300174ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/openbsd-amd64-70/2: 7 files, 2 dirs (7.084985ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/freebsd-amd64-13_0/7: 7 files, 2 dirs (7.235414ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/windows-386-2008-newcc/4: 7 files, 2 dirs (4.711504ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/linux-amd64-buster/0: 7 files, 2 dirs (8.554855ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/freebsd-386-12_3/1: 7 files, 2 dirs (10.617952ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/linux-amd64-boringcrypto/8: 7 files, 2 dirs (5.253961ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/windows-amd64-longtest-newcc/9: 7 files, 2 dirs (5.336727ms)
2022/09/08 07:14:42 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/windows-386-2012/11: 7 files, 2 dirs (5.40824ms)
...
2022/09/08 07:14:45 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/linux-amd64-stretch/69/go1.4: 1 files, 1 dirs (1.413401ms)
2022/09/08 07:14:45 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/darwin-amd64-12_0/71: 11 files, 7 dirs (5.096348ms)
2022/09/08 07:14:45 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta3649961871/001/darwin-amd64-12_0/71/go1.4: 1 files, 1 dirs (555.216µs)
panic: test timed out after 10m0s

testing.(*T).Run(0xc0004fe9c0, {0x1e28bbf?, 0x6319cea2?}, 0x1ec8228)
	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/go/src/testing/testing.go:1527 +0x37a
golang.org/x/build/internal/relui.TestRelease(0x7?)
	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:43 +0x31
testing.tRunner(0xc0004fe9c0, 0x1ec8238)
2022-09-08 09:47 linux-amd64 build@b4daf1e8 go@cdc6aeb6 golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/darwin-arm64-12/0: 7 files, 2 dirs (1.439685ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/windows-amd64-2016/3: 7 files, 2 dirs (1.374886ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/freebsd-386-13_0/6: 7 files, 2 dirs (3.539156ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/windows-386-2008/7: 7 files, 2 dirs (4.01809ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/linux-amd64-buster/2: 7 files, 2 dirs (1.674ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/windows-amd64-newcc-race/10: 7 files, 2 dirs (1.421991ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/linux-amd64-bullseye/1: 7 files, 2 dirs (424.895µs)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/linux-386/5: 7 files, 2 dirs (1.729743ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/darwin-arm64-12/0/go1.4: 1 files, 1 dirs (1.609308ms)
2022/09/08 09:48:58 extracted tarball into /workdir/tmp/TestReleasebeta1625751918/001/windows-386-2012-newcc/4: 7 files, 2 dirs (732.132µs)
...
2022/09/08 09:48:59 extracted tarball into /workdir/tmp/TestReleaserc612727468/001/linux-amd64-stretch/71/go1.4: 1 files, 1 dirs (113.486µs)
panic: test timed out after 10m0s

golang.org/x/build/internal/workflow.(*Workflow).Run(0xc000441640, {0x1465ff8?, 0xc000441000}, {0x1460480, 0xc000e1e370})
	/workdir/gopath/src/golang.org/x/build/internal/workflow/workflow.go:767 +0x318
golang.org/x/build/internal/relui.testRelease(0xc00084ba00, {0x1232a82, 0x9}, 0x2)
	/workdir/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:186 +0x3ef
golang.org/x/build/internal/relui.TestRelease.func2(0xc00084b520?)
	/workdir/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:47 +0x2a
testing.tRunner(0xc00084ba00, 0x12c83c0)
2022-09-08 14:53 darwin-amd64-10_15 build@656fd833 go@0e50bf0e golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/windows-amd64-longtest-newcc/15: 7 files, 2 dirs (4.344228ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-amd64-noopt/14: 7 files, 2 dirs (5.064686ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/windows-amd64-2008/16: 7 files, 2 dirs (6.092823ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-386-softfloat/12: 7 files, 2 dirs (8.604771ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-386-clang/4: 7 files, 2 dirs (12.325998ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-arm-aws/13: 7 files, 2 dirs (9.826523ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-amd64-stretch/3: 7 files, 2 dirs (14.030131ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/linux-amd64-fedora/5: 7 files, 2 dirs (4.763252ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/windows-amd64-race/1: 7 files, 2 dirs (2.810633ms)
2022/09/08 14:19:54 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/darwin-arm64-12/18: 7 files, 2 dirs (6.928708ms)
...
2022/09/08 14:19:56 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/freebsd-amd64-12_3/70/go1.4: 1 files, 1 dirs (492.864µs)
2022/09/08 14:19:57 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/darwin-amd64-12_0/71: 11 files, 7 dirs (3.250036ms)
2022/09/08 14:19:57 extracted tarball into /var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/tmp/TestReleasebeta2029754536/001/darwin-amd64-12_0/71/go1.4: 1 files, 1 dirs (460.418µs)
panic: test timed out after 10m0s

testing.(*T).Run(0xc0004ccb60, {0x1e28ddf?, 0x631a324a?}, 0x1ec8448)
	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/go/src/testing/testing.go:1527 +0x37a
golang.org/x/build/internal/relui.TestRelease(0x7?)
	/var/folders/kh/5zzynz152r94t18yzstnrwx80000gn/T/workdir-host-darwin-amd64-10_15/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:43 +0x31
testing.tRunner(0xc0004ccb60, 0x1ec8458)
2022-09-08 14:53 darwin-amd64-12_0 build@656fd833 go@218294f1 golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/linux-amd64-noopt/3: 7 files, 2 dirs (8.70321ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/windows-amd64-longtest-newcc/11: 7 files, 2 dirs (3.702371ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/linux-amd64-sid/14: 7 files, 2 dirs (3.164109ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/android-amd64-emu/10: 7 files, 2 dirs (3.938059ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/windows-amd64-newcc-race/2: 7 files, 2 dirs (13.77306ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/netbsd-386-9_0/8: 7 files, 2 dirs (6.18852ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/linux-amd64-stretch/12: 7 files, 2 dirs (7.58989ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/linux-amd64-ssacheck/13: 7 files, 2 dirs (4.195361ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/openbsd-386-70/5: 7 files, 2 dirs (6.363967ms)
2022/09/09 06:34:56 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/linux-amd64-nocgo/4: 7 files, 2 dirs (14.469547ms)
...
2022/09/09 06:35:01 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/windows-amd64-2008/70/go1.4: 1 files, 1 dirs (1.785294ms)
2022/09/09 06:35:01 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/darwin-arm64-12/71: 11 files, 7 dirs (6.398021ms)
2022/09/09 06:35:01 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta1036925895/001/darwin-arm64-12/71/go1.4: 1 files, 1 dirs (577.849µs)
panic: test timed out after 10m0s

testing.(*T).Run(0xc0003ca680, {0x1e28d5f?, 0x631b4100?}, 0x1ec83c8)
	/Users/gopher/workdir/go/src/testing/testing.go:1527 +0x37a
golang.org/x/build/internal/relui.TestRelease(0x7?)
	/Users/gopher/workdir/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:43 +0x31
testing.tRunner(0xc0003ca680, 0x1ec83d8)
2022-09-08 14:53 darwin-amd64-12_0 build@656fd833 go@2c3187cd golang.org/x/build/internal/relui.TestRelease (log) 
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/openbsd-amd64-68/1: 7 files, 2 dirs (4.067404ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-amd64-2016/5: 7 files, 2 dirs (4.528866ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/darwin-arm64-12/13: 7 files, 2 dirs (3.48113ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-386-2012/4: 7 files, 2 dirs (5.925189ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/linux-amd64-boringcrypto/14: 7 files, 2 dirs (4.951412ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/darwin-arm64-12/13/go1.4: 1 files, 1 dirs (521.51µs)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-amd64-2016/5/go1.4: 1 files, 1 dirs (615µs)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/netbsd-amd64-9_0/15: 7 files, 2 dirs (8.848539ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/openbsd-amd64-68/1/go1.4: 1 files, 1 dirs (1.589824ms)
2022/09/08 10:18:00 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/linux-amd64-boringcrypto/14/go1.4: 1 files, 1 dirs (1.615996ms)
...
2022/09/08 10:18:04 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-amd64-2008/71: 11 files, 7 dirs (10.789719ms)
2022/09/08 10:18:04 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-amd64-2008/70/go1.4: 1 files, 1 dirs (814.186µs)
2022/09/08 10:18:04 extracted tarball into /Users/gopher/workdir/tmp/TestReleasebeta3550069186/001/windows-amd64-2008/71/go1.4: 1 files, 1 dirs (2.698529ms)
panic: test timed out after 10m0s

testing.(*T).Run(0xc00038e1a0, {0x1e28dbf?, 0x631a23c8?}, 0x1ec8428)
	/Users/gopher/workdir/go/src/testing/testing.go:1527 +0x37a
golang.org/x/build/internal/relui.TestRelease(0x7?)
	/Users/gopher/workdir/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:43 +0x31
testing.tRunner(0xc00038e1a0, 0x1ec8438)

— watchflakes

@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/build/internal/relui" && test == "TestRelease"
2022-09-20 15:11 linux-amd64 build@577b49ce go@1eeb257b x/build/internal/relui.TestRelease (log) 
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-amd64-ssacheck/9: 7 files, 2 dirs (979.447µs)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-386-stretch/2: 7 files, 2 dirs (3.018023ms)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/windows-arm64-11/4: 7 files, 2 dirs (740.171µs)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/freebsd-amd64-13_0/3: 7 files, 2 dirs (3.077984ms)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/windows-386-2012-newcc/16: 7 files, 2 dirs (1.846636ms)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-amd64-clang/1: 7 files, 2 dirs (4.05206ms)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/android-386-emu/19: 7 files, 2 dirs (490.606µs)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-amd64-stretch/17: 7 files, 2 dirs (2.432913ms)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-386-stretch/2/go1.4: 1 files, 1 dirs (117.811µs)
2022/09/20 15:13:25 extracted tarball into /workdir/tmp/TestReleasebeta1320469778/001/linux-amd64-staticlockranking/21: 7 files, 2 dirs (381.686µs)
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0x12859d4?, 0x6?})
	/workdir/go/src/runtime/panic.go:1047 +0x5d fp=0x7fc0557f9ca8 sp=0x7fc0557f9c78 pc=0x4387fd
runtime.badPointer(0x7fc0745f34b8, 0x7fc0557f9d20?, 0xc002164000, 0x7fc0557f9db0?)
	/workdir/go/src/runtime/mbitmap.go:314 +0x150 fp=0x7fc0557f9cf8 sp=0x7fc0557f9ca8 pc=0x4163b0
runtime.findObject(0xc00112a000?, 0xc000640148?, 0x0?)
	/workdir/go/src/runtime/mbitmap.go:357 +0xa6 fp=0x7fc0557f9d30 sp=0x7fc0557f9cf8 pc=0x416546
runtime.scanobject(0xc00005e638?, 0xc00005e638)
...
	/workdir/go/src/runtime/proc.go:375 +0xd6 fp=0xc000083de0 sp=0xc000083dc0 pc=0x43b4b6
runtime.chanrecv(0xc0003600e0, 0xc000083edf, 0x1)
	/workdir/go/src/runtime/chan.go:583 +0x49d fp=0xc000083e70 sp=0xc000083de0 pc=0x408a1d
runtime.chanrecv1(0xc00046e980?, 0x1?)
	/workdir/go/src/runtime/chan.go:442 +0x18 fp=0xc000083e98 sp=0xc000083e70 pc=0x408518
testing.(*T).Run(0xc0001029c0, {0x1223687?, 0xc000083f40?}, 0x12c3598)
	/workdir/go/src/testing/testing.go:1529 +0x37a fp=0xc000083f38 sp=0xc000083e98 pc=0x525d9a
golang.org/x/build/internal/relui.TestRelease(0x0?)
	/workdir/gopath/src/golang.org/x/build/internal/relui/buildrelease_test.go:42 +0x31 fp=0xc000083f70 sp=0xc000083f38 pc=0xf5ac51
testing.tRunner(0xc0001029c0, 0x12c35a8)

— watchflakes

@dmitshur
Copy link
Contributor

This seems to be the tracking the same problem as issue #53972 (CC @heschi). We think this might be fixed as of Sept 9 (see #53972 (comment)).

The finding above from Sept 20 has "fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)" which is odd.

@dmitshur
Copy link
Contributor

Closing in favor of #53972.

@dmitshur dmitshur closed this as not planned Won't fix, can't repro, duplicate, stale Sep 27, 2022
@gopherbot gopherbot reopened this Nov 4, 2022
@gopherbot
Copy link
Author

Found new dashboard test flakes for:

#!watchflakes
post <- pkg == "golang.org/x/build/internal/relui" && test == "TestRelease"
2022-11-04 18:55 darwin-arm64-12 build@c8c8e67e go@5d5ed57b x/build/internal/relui.TestRelease (log) 
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/linux-s390x-crosscompile/0: 7 files, 2 dirs (1.231916ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/openbsd-386-68/1: 7 files, 2 dirs (1.537542ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/linux-amd64-race/5: 7 files, 2 dirs (2.592542ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/windows-386-2008/3: 7 files, 2 dirs (3.619667ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/openbsd-386-70/4: 7 files, 2 dirs (3.424875ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/linux-s390x-crosscompile/0/go1.4: 1 files, 1 dirs (267.166µs)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/openbsd-amd64-70/2: 7 files, 2 dirs (4.705958ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/openbsd-386-68/1/go1.4: 1 files, 1 dirs (1.93575ms)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/openbsd-amd64-70/2/go1.4: 1 files, 1 dirs (746.417µs)
2022/11/04 15:00:41 extracted tarball into /tmp/buildlet/tmp/TestReleasebeta3844974960/002/windows-386-2008/3/go1.4: 1 files, 1 dirs (766.916µs)
...
--- FAIL: TestRelease (3.53s)
    --- FAIL: TestRelease/rc (1.51s)
        buildrelease_test.go:660: task Run advisory TryBot darwin-amd64-10_15: started
        buildrelease_test.go:660: task windows-amd64: Build MSI installer: started
        buildrelease_test.go:660: task Upload artifacts to CDN: started
        buildrelease_test.go:660: task linux-386: Build binary archive: started
        buildrelease_test.go:660: task Run advisory TryBot js-wasm: started
        buildrelease_test.go:660: task linux-s390x: Build binary archive: started
        buildrelease_test.go:660: task Run advisory TryBot linux-386-bullseye: started
        buildrelease_test.go:660: task windows-amd64: Convert to .zip: started
...
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/linux-amd64-stretch/64: 11 files, 7 dirs (1.355125ms)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/linux-amd64-stretch/63: 11 files, 7 dirs (1.902166ms)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/linux-amd64-stretch/64/go1.4: 1 files, 1 dirs (208.458µs)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/linux-amd64-stretch/63/go1.4: 1 files, 1 dirs (326.209µs)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-amd64-12_0/66: 11 files, 7 dirs (7.531291ms)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-amd64-12_0/66/go1.4: 1 files, 1 dirs (388.458µs)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-amd64-12_0/65/pkg-root/usr/local: 11 files, 7 dirs (2.518542ms)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-arm64-12/67: 11 files, 7 dirs (1.915291ms)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-arm64-12/67/go1.4: 1 files, 1 dirs (305.75µs)
2022/11/04 15:00:49 extracted tarball into /tmp/buildlet/tmp/TestAdvisoryTrybotFail3084135081/002/darwin-arm64-12/68/pkg-root/usr/local: 11 files, 7 dirs (1.468375ms)

— watchflakes

@dmitshur dmitshur self-assigned this Nov 4, 2022
@dmitshur dmitshur added the Testing An issue that has been verified to require only test changes, not just a test failure. label Nov 4, 2022
@gopherbot
Copy link
Author

Change https://go.dev/cl/448135 mentions this issue: internal/relui: wait for all files in uploadArtifacts

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 4, 2022
@golang golang locked and limited conversation to collaborators Nov 7, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@dmitshur dmitshur

Labels
Builders x/build issues (builders, bots, dashboards) FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
Go Release
Archived in project
Test Flakes
Archived in project
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@rsc @dmitshur @gopherbot