Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

os,internal/poll: reads from closed pipes sometimes do not unblock on darwin #63937

Open
bcmills opened this issue Nov 3, 2023 · 24 comments
Open

os,internal/poll: reads from closed pipes sometimes do not unblock on darwin #63937

bcmills opened this issue Nov 3, 2023 · 24 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Milestone
Go1.23

Comments

@bcmills
Copy link
Contributor

bcmills commented Nov 3, 2023 •
edited by cherrymui 

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")

We have been tracking a long-running bug on macOS (darwin) in #54461 (comment). That issue title doesn't accurately capture the widespread nature of the problem, nor its defining symptom or suspected root cause; so, I am filing a new issue for it.

On macOS, we see a pattern of test failures with the following characteristics:

Many of these bugs have been worked around by explicitly canceling (or adding a timeout to) the stuck read.

The issues with this symptom have included:

A few other issues may be related but don't (or didn't) include goroutine dumps for the stuck process: they may or may not be due to this failure mode.

Given the symptoms, it seems likely to me that this is either a bug in internal/poll, or a bug in the macOS platform that we have somehow started to trigger.

Note that #61779 was bisected specifically to https://go.dev/cl/420334.
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Nov 3, 2023
This was referenced Nov 3, 2023
Open
Closed
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 3, 2023
@dmitshur dmitshur added this to the Go1.22 milestone Nov 3, 2023
@bcmills bcmills mentioned this issue Nov 8, 2023
Open
@mknyszek
Copy link
Contributor

mknyszek commented Nov 8, 2023

In triage, we think maybe this and the darwin memory corruption issues would be a good target for our friction fixit week time. We think the next step is to attempt to reproduce the issue in C and see if we need to escalate this as a kernel bug or something.

This was referenced Nov 10, 2023
Closed
Open
@bcmills
Copy link
Contributor Author

bcmills commented Nov 15, 2023 •
edited

I randomly stumbled across #25696 today.

It isn't clear to me whether that is related, since it is using net pipes instead of os pipes. 🤔

@bcmills bcmills mentioned this issue Nov 20, 2023
Closed
This was referenced Dec 1, 2023
Open
Open
@bcmills bcmills mentioned this issue Dec 13, 2023
Open
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2023-12-15 21:38 darwin-arm64-11 tools@3bed1477 go@8cb86b5f x/tools/gopls/internal/cmd.TestExecute (log) 
2023/12/15 16:40:41 New session
	create_session=1
Log: Loading packages...
2023/12/15 16:40:41 go info for /tmp/buildlet/tmp/fake708580075
(go dir /tmp/buildlet/tmp/fake708580075)
(go version go version go1.20.12 darwin/arm64)
(valid build configuration = true)
(build flags: [])
(selected go env: [GO111MODULE=, GOCACHE=/tmp/buildlet/gocache, GOFLAGS=, GOMODCACHE=/tmp/buildlet/gopath/pkg/mod, GOPATH=/tmp/buildlet/gopath, GOPRIVATE=, GOROOT=/tmp/buildlet/go, GOWORK=])

...
panic: test timed out after 10m0s
running tests:
	TestExecute (9m59s)

os/exec.(*Cmd).awaitGoroutines(0x140001a3ce0, 0x0)
	/tmp/buildlet/go/src/os/exec/exec.go:941 +0x174
os/exec.(*Cmd).Wait(0x140001a3ce0)
	/tmp/buildlet/go/src/os/exec/exec.go:908 +0x164
os/exec.(*Cmd).Run(0x140006be380?)
	/tmp/buildlet/go/src/os/exec/exec.go:590 +0x38
golang.org/x/tools/gopls/internal/cmd_test.goplsWithEnv(0x1400033a680, {0x14000504240, 0x33}, {0x0, 0x0, 0x102b0878f?}, {0x1400052be48, 0x3, 0x3})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1085 +0x324
golang.org/x/tools/gopls/internal/cmd_test.gopls(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1066
golang.org/x/tools/gopls/internal/cmd_test.TestExecute(0x0?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:338 +0x540
testing.tRunner(0x1400033a680, 0x102f7c470)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2023-12-07 19:37 darwin-amd64-11_0 pkgsite@00698da1 go@9b4b3e5a (log) 
runtime: pointer 0xc000ced4b9 to unused region of span span.base()=0xc000cea000 span.limit=0xc000cf9f00 span.state=1
runtime: found in object at *(0xc001265d60+0x0)
object=0xc001265d60 s.base()=0xc001264000 s.limit=0xc001265fe0 s.spanclass=14 s.elemsize=80 s.state=mSpanInUse
 *(object+0) = 0xc000ced4b9 <==
 *(object+8) = 0x6
 *(object+16) = 0xc0004047e8
 *(object+24) = 0x8
 *(object+32) = 0xc00133e0a0
 *(object+40) = 0x1d
 *(object+48) = 0xc0000280af
...
fatal error: found bad pointer in Go heap (incorrect use of unsafe or cgo?)

runtime stack:
runtime.throw({0xf8793f0?, 0x6?})
	runtime/panic.go:1023 +0x5c fp=0x700008ed5d88 sp=0x700008ed5d58 pc=0xf2f73dc
runtime.badPointer(0x16cb2980, 0xc000ced4b9, 0xc001265d60, 0x0)
	runtime/mbitmap.go:286 +0x165 fp=0x700008ed5dd8 sp=0x700008ed5d88 pc=0xf2d4e05
runtime.findObject(0xc00002a080?, 0xc001265d60?, 0xf2e8545?)
	runtime/mbitmap.go:329 +0xa6 fp=0x700008ed5e10 sp=0x700008ed5dd8 pc=0xf2d4f86
runtime.scanobject(0xc00003f768?, 0xc00003f768)
...
os/exec.(*Cmd).writerDescriptor.func1()
	os/exec/exec.go:577 +0x34 fp=0xc001a85f60 sp=0xc001a85f00 pc=0xf3e04f4
os/exec.(*Cmd).Start.func2(0xc0005382c0?)
	os/exec/exec.go:724 +0x2c fp=0xc001a85fc8 sp=0xc001a85f60 pc=0xf3e118c
os/exec.(*Cmd).Start.gowrap1()
	os/exec/exec.go:736 +0x24 fp=0xc001a85fe0 sp=0xc001a85fc8 pc=0xf3e1124
runtime.goexit({})
	runtime/asm_amd64.s:1695 +0x1 fp=0xc001a85fe8 sp=0xc001a85fe0 pc=0xf331c21
created by os/exec.(*Cmd).Start in goroutine 7451
	os/exec/exec.go:723 +0x9ab

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2023-12-18 20:56 darwin-amd64-13 tools@ee47fd47 go@c564d4ae (log)

— watchflakes

@bcmills
Copy link
Contributor Author

bcmills commented Jan 3, 2024

I think this is caused by an incomplete fix for #24164. We avoid adding FDs of type S_IFIFO to the poller if they are marked as kindOpenFile, but the FDs opened by os.Pipe are marked as kindPipe instead (even though they are also of type S_IFIFO — see https://go.dev/play/p/6kSh4pJj7ef).

This is probably also related to the difference in unix.Select behavior recently reported in https://groups.google.com/g/golang-nuts/c/wzh-nwH9dH8.

@panjf2000
Copy link
Member

panjf2000 commented Jan 4, 2024 •
edited

I'm not sure I agree with your conclusion.

Despite the similarity in inter-process communication, Pipes and FIFOs are fundamentally different: Pipes are created in memory, and their existence is transient while FIFOs persist as special files in the file system until they are removed. And kqueue works according to the manual when it comes to Pipes:

When the last writer disconnects, the filter will set EV_EOF in flags. This may be cleared by passing in EV_CLEAR, at which point the filter will resume waiting ing for data to become available before returning.

The filter will set EV_EOF when the reader disconnects, and for the fifo case, this may be cleared by use of EV_CLEAR. Note that this filter is not supported for vnodes.

, which is why TestPipeEOF has been succeeding.

The only problem is FIFO, kqueue doesn't seem to comply with its manuals when working with FIFOs, I assume that has something to do with the fact that FIFOs persist as special files in the file system. Thus, I'm inclined to blame this flake on macOS kernel.
@bcmills

@drakkan drakkan mentioned this issue Jan 4, 2024
Open
@bcmills
Copy link
Contributor Author

bcmills commented Jan 4, 2024

Huh. Looking a bit more closely:

So it seems plausible that either this is a composite of more than one bug, or perhaps a general race condition in the poller (not specific to pipes).

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2024-01-05 15:13 darwin-amd64-nocgo tools@920d665b go@8db13108 x/tools/gopls/internal/test/marker.Test (log) 
panic: test timed out after 10m0s
running tests:
	Test/codeaction/extract_variable.txt (6s)
	Test/codeaction/fill_struct.txt (1s)
	Test/codeaction/functionextraction.txt (4s)
	Test/codeaction/imports.txt (6s)
	Test/codeaction/invertif.txt (29s)
	Test/links/links.txt (44s)

golang.org/x/tools/gopls/internal/test/integration.(*Awaiter).Await(0xc02d790d20, {0x87be678, 0xc00f458390}, {0xc009e88a80, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/integration/env.go:374 +0x3c9
golang.org/x/tools/gopls/internal/test/marker.newEnv(0xc000160b60, 0xc000630a60, 0xc000397a70, 0x0, {0x0, 0x0, 0xc0000fb0e0?}, {{0x0, 0x0}, 0x0, ...})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/marker/marker_test.go:800 +0x96e
golang.org/x/tools/gopls/internal/test/marker.Test.func1(0xc000160b60)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/marker/marker_test.go:140 +0x44e
testing.tRunner(0xc000160b60, 0xc000286888)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2024-01-05 14:42 darwin-amd64-nocgo tools@88ea935e go@8db13108 x/tools/gopls/internal/test/marker.Test (log) 
panic: test timed out after 10m0s
running tests:
	Test/codeaction/extract_variable.txt (7s)
	Test/codeaction/functionextraction.txt (7s)
	Test/codeaction/imports.txt (10s)
	Test/codeaction/invertif.txt (31s)
	Test/codeaction/removeparam.txt (38s)
	Test/completion/foobarbaz.txt (1m57s)

golang.org/x/tools/gopls/internal/test/integration.(*Awaiter).Await(0xc0289f4540, {0xb88a778, 0xc0060158c0}, {0xc02828cf90, 0x1, 0x1})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/integration/env.go:374 +0x3c9
golang.org/x/tools/gopls/internal/test/marker.newEnv(0xc0006f8680, 0xc0005d6480, 0xc00021ede0, 0x0, {0x0, 0x0, 0xc0001fe140?}, {{0x0, 0x0}, 0x0, ...})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/marker/marker_test.go:800 +0x96e
golang.org/x/tools/gopls/internal/test/marker.Test.func1(0xc0006f8680)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/test/marker/marker_test.go:140 +0x44e
testing.tRunner(0xc0006f8680, 0xc000122498)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2024-01-05 15:29 darwin-amd64-11_0 tools@3e8a0a3b go@5c38c049 x/tools/gopls/internal/cmd.TestCodeLens (log) 
2024/01/05 11:05:23 New session
	create_session=1
Log: Loading packages...
2024/01/05 11:05:23 go info for /tmp/buildlet/tmp/fake2554693110
(view type GoModView)
(root dir /tmp/buildlet/tmp/fake2554693110)
(go version go version go1.20.12 darwin/amd64)
(build flags: [])
(go env: {GOOS:darwin GOARCH:amd64 GOCACHE:/tmp/buildlet/gocache GOMODCACHE:/tmp/buildlet/gopath/pkg/mod GOPATH:/tmp/buildlet/gopath GOPRIVATE: GOFLAGS: GO111MODULE: GoVersion:20 GoVersionOutput:go version go1.20.12 darwin/amd64
 GOWORK: GOPACKAGESDRIVER:})
...
panic: test timed out after 10m0s
running tests:
	TestCodeLens (9m58s)

os/exec.(*Cmd).awaitGoroutines(0xc000656160, 0x0)
	/tmp/buildlet/go/src/os/exec/exec.go:941 +0x233
os/exec.(*Cmd).Wait(0xc000656160)
	/tmp/buildlet/go/src/os/exec/exec.go:908 +0x175
os/exec.(*Cmd).Run(0xc0000363c0?)
	/tmp/buildlet/go/src/os/exec/exec.go:590 +0x39
golang.org/x/tools/gopls/internal/cmd_test.goplsWithEnv(0xc00068bd40, {0xc0006e0280, 0x34}, {0x0, 0x0, 0x11316f3?}, {0xc0003d4ee0, 0x4, 0x4})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1085 +0x370
golang.org/x/tools/gopls/internal/cmd_test.gopls(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1066
golang.org/x/tools/gopls/internal/cmd_test.TestCodeLens(0x4?)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:206 +0x326
testing.tRunner(0xc00068bd40, 0x1d6fca8)

— watchflakes

@bcmills bcmills mentioned this issue Jan 5, 2024
Open
@panjf2000
Copy link
Member

panjf2000 commented Jan 8, 2024 •
edited

I think this is caused by an incomplete fix for #24164. We avoid adding FDs of type S_IFIFO to the poller if they are marked as kindOpenFile, but the FDs opened by os.Pipe are marked as kindPipe instead (even though they are also of type S_IFIFO — see https://go.dev/play/p/6kSh4pJj7ef).

This is probably also related to the difference in unix.Select behavior recently reported in https://groups.google.com/g/golang-nuts/c/wzh-nwH9dH8.

Although a new patch that stops pipes being added into netpoll like #24164 and the rationale behind it can't explain the bug here on the basis of my comment, but before we figure out what really happened with pipes in kqueue, something like that can be a temporary solution that's a good chance to make these test flakes go away becausekqueue doesn't look like to be stable when it comes to pipes.

I can draft a CL, in which we knock off pipes from netpoll on macOS and leave a TODO comment if we eventually decide to go for it.

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2024-01-08 23:16 darwin-amd64-12_0 tools@c95fa0ff go@b7c630dc x/tools/gopls/internal/cmd.TestFoldingRanges (log) 
2024/01/08 18:52:56 New session
	create_session=1
Log: Loading packages...
2024/01/08 18:52:56 go info for /tmp/buildlet/tmp/fake544103524
(view type GoModView)
(root dir /tmp/buildlet/tmp/fake544103524)
(go version go version devel b7c630dc3ac3f43b2294f803f26f512d75a54fc6 darwin/amd64)
(build flags: [])
(go env: {GOOS:darwin GOARCH:amd64 GOCACHE:/tmp/buildlet/gocache GOMODCACHE:/tmp/buildlet/gopath/pkg/mod GOPATH:/tmp/buildlet/gopath GOPRIVATE: GOFLAGS: GO111MODULE: GoVersion:22 GoVersionOutput:go version devel b7c630dc3ac3f43b2294f803f26f512d75a54fc6 darwin/amd64
 GOWORK: GOPACKAGESDRIVER:})
...
panic: test timed out after 10m0s
running tests:
	TestFoldingRanges (9m57s)

os/exec.(*Cmd).awaitGoroutines(0xc000420160, 0x0)
	/tmp/buildlet/go/src/os/exec/exec.go:948 +0x1fa
os/exec.(*Cmd).Wait(0xc000420160)
	/tmp/buildlet/go/src/os/exec/exec.go:915 +0x16c
os/exec.(*Cmd).Run(0xc000420160)
	/tmp/buildlet/go/src/os/exec/exec.go:607 +0x2d
golang.org/x/tools/gopls/internal/cmd_test.goplsWithEnv(0xc00048e1a0, {0xc00054e180, 0x39}, {0x0, 0x0, 0xc000625720?}, {0xc00058cf30, 0x2, 0x2})
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1085 +0x33e
golang.org/x/tools/gopls/internal/cmd_test.gopls(...)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:1066
golang.org/x/tools/gopls/internal/cmd_test.TestFoldingRanges(0xc00048e1a0)
	/tmp/buildlet/gopath/src/golang.org/x/tools/gopls/internal/cmd/integration_test.go:368 +0x12c
testing.tRunner(0xc00048e1a0, 0x5abc670)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead`
2024-01-08 18:46 darwin-amd64-12_0 build@21e06540 go@59ecebf4 (log)

— watchflakes

@gopherbot

This comment was marked as off-topic.

Sign in to view
@bcmills bcmills mentioned this issue Jan 23, 2024
Open
@gopherbot gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024
@bcmills bcmills mentioned this issue Feb 16, 2024
Open
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-02-21 22:16 gotip-darwin-amd64-longtest go@507d1b22 cmd/go.TestScript/repro_build (log) 
=== RUN   TestScript/repro_build
=== PAUSE TestScript/repro_build
=== CONT  TestScript/repro_build
    script_test.go:132: 2024-02-22T00:18:14Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-4052389089/tmpdir991360422/repro_build1986575617
2024/02/21 16:18:15 serving /git/hello/info/refs?service=git-upload-pack
2024/02/21 16:18:16 hello.txt:
> handle git
> env GIT_AUTHOR_NAME=bwk
> env GIT_AUTHOR_EMAIL=bwk
...
        r13    0x0
        r14    0xc000585a40
        r15    0x1fffeffdf9b
        rip    0x7ff8138b9a60
        rflags 0x246
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:156: FAIL: testdata/script/repro_build.txt:11: go build -a -o http18.o net/http: exit status 2
--- FAIL: TestScript/repro_build (95.12s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-02-26 20:48 gotip-darwin-amd64-longtest go@cab90c84 cmd/go.TestScript/test_fuzz (log) 
=== RUN   TestScript/test_fuzz
=== PAUSE TestScript/test_fuzz
=== CONT  TestScript/test_fuzz
    script_test.go:132: 2024-02-27T12:09:03Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-937896419/tmpdir3840417651/test_fuzz1080357726
    script_test.go:156: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-937896419/tmpdir3840417651/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        --- FAIL: Fuzz (0.00s)
            fuzz.go:652: returned without calling F.Fuzz, F.Fail, or F.Skip
        FAIL
        FAIL	command-line-arguments	0.019s
        FAIL
        [exit status 1]
        > ! stdout ^ok
        > stdout FAIL
        matched: --- FAIL: Fuzz (0.00s)
        # Test that fuzzing a fuzz target that returns without failing or calling
...
        r13    0x0
        r14    0xc0000081c0
        r15    0x0
        rip    0x641a980
        rflags 0x10246
        cs     0x2b
        fs     0x0
        gs     0x0
    script_test.go:156: FAIL: testdata/script/test_fuzz.txt:193: go test -run FuzzInNestedDir corpustesting/fuzz_testdata_corpus_test.go: exit status 2
--- FAIL: TestScript/test_fuzz (139.59s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-02-27 18:08 gotip-darwin-amd64-longtest go@296d0812 cmd/go.TestScript/cover_pkgall_runtime (log) 
=== RUN   TestScript/cover_pkgall_runtime
=== PAUSE TestScript/cover_pkgall_runtime
=== CONT  TestScript/cover_pkgall_runtime
    script_test.go:132: 2024-02-28T16:11:29Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-425658779/tmpdir1321256165/cover_pkgall_runtime3700749184
    script_test.go:156: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-425658779/tmpdir1321256165/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x1000100010100
        r14    0x7ff85512ce80
        r15    0x10000
        rip    0x7ff811b175d6
        rflags 0x247
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:156: FAIL: testdata/script/cover_pkgall_runtime.txt:12: go test -coverpkg=all -race x: exit status 2
--- FAIL: TestScript/cover_pkgall_runtime (91.06s)
2024-02-27 21:07 gotip-darwin-amd64-longtest go@b847d4cd cmd/go.TestScript/repro_build (log) 
=== RUN   TestScript/repro_build
=== PAUSE TestScript/repro_build
=== CONT  TestScript/repro_build
    script_test.go:132: 2024-02-28T11:14:45Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-2708383841/tmpdir3975560248/repro_build665426348
2024/02/28 03:14:45 serving /git/hello/git-upload-pack
2024/02/28 03:14:46 serving /git/hello/info/refs?service=git-upload-pack
2024/02/28 03:14:46 serving /git/hello/git-upload-pack
2024/02/28 03:14:46 serving /git/hello/git-upload-pack
2024/02/28 03:14:49 serving /git/hello/info/refs?service=git-upload-pack
...
        r13    0x3b0100003c00
        r14    0x7ff85b6ece80
        r15    0x1900
        rip    0x7ff8180d75d6
        rflags 0x247
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:156: FAIL: testdata/script/repro_build.txt:8: go build -a -o http17.o net/http: exit status 2
--- FAIL: TestScript/repro_build (65.76s)
2024-02-28 18:17 gotip-darwin-amd64-longtest go@2278ffcb cmd/go.TestScript/test_rebuildall (log) 
=== RUN   TestScript/test_rebuildall
=== PAUSE TestScript/test_rebuildall
=== CONT  TestScript/test_rebuildall
    script_test.go:132: 2024-02-28T20:24:55Z
    script_test.go:134: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-4170341431/tmpdir3229471862/test_rebuildall1078707148
    script_test.go:156: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-4170341431/tmpdir3229471862/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x1
        r14    0x78e7060
        r15    0x1
        rip    0x7ff80e1e2706
        rflags 0x203
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:156: FAIL: testdata/script/test_rebuildall.txt:8: go test -x -a -c testdata/dep_test.go: exit status 2
--- FAIL: TestScript/test_rebuildall (28.65s)
2024-02-28 18:17 gotip-darwin-amd64-longtest go@2278ffcb runtime.TestPingPongHog (log) 
=== RUN   TestPingPongHog
    proc_test.go:490: want hogCount/lightCount in [0.05, 20]; got 50000000/1636000 = 30.56234718826406
--- FAIL: TestPingPongHog (0.11s)

— watchflakes

@panjf2000 panjf2000 mentioned this issue Mar 11, 2024
Closed
@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-03-25 21:28 x_tools-gotip-darwin-amd64_11 tools@2c8dd3ec go@e7bdc881 x/tools/gopls/internal/test/integration/diagnostics.TestInitialization [ABORT] (log) 
=== RUN   TestInitialization
2024-03-25 21:28 x_tools-gotip-darwin-amd64_11 tools@2c8dd3ec go@e7bdc881 x/tools/gopls/internal/test/integration/misc.TestRenamePackageWithConflicts [ABORT] (log) 
=== RUN   TestRenamePackageWithConflicts
2024-03-25 21:28 x_tools-gotip-darwin-amd64_11 tools@2c8dd3ec go@e7bdc881 x/tools/gopls/internal/test/marker.Test [ABORT] (log) 
=== RUN   Test

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-03-26 19:50 gotip-darwin-amd64-longtest go@51a96f86 cmd/go.TestScript/test_vet (log) 
=== RUN   TestScript/test_vet
=== PAUSE TestScript/test_vet
=== CONT  TestScript/test_vet
    script_test.go:136: 2024-03-26T21:44:30Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-3443777511/tmpdir933495839/test_vet2204836050
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-3443777511/tmpdir933495839/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x1e0100001f00
        r14    0x1f00
        r15    0x1e00
        rip    0x7ff813f9b5d6
        rflags 0x297
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:160: FAIL: testdata/script/test_vet.txt:32: go test -vet=atomic,bools,nilfunc ./vetall/...: exit status 2
--- FAIL: TestScript/test_vet (24.32s)
2024-03-27 21:58 gotip-darwin-amd64-longtest go@2e1003e2 cmd/go.TestScript/build_trimpath_cgo (log) 
=== RUN   TestScript/build_trimpath_cgo
=== PAUSE TestScript/build_trimpath_cgo
=== CONT  TestScript/build_trimpath_cgo
    script_test.go:136: 2024-03-28T00:03:28Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1676868827/tmpdir844700615/build_trimpath_cgo3717282694
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1676868827/tmpdir844700615/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x220100002300
        r14    0x7ff858645e80
        r15    0x2200
        rip    0x7ff8150305d6
        rflags 0x247
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:160: FAIL: testdata/script/build_trimpath_cgo.txt:20: go run ./list-dwarf hello.exe: exit status 2
--- FAIL: TestScript/build_trimpath_cgo (24.96s)
2024-03-29 16:41 gotip-darwin-amd64_11 go@8f59b28f cmd/cgo/internal/testplugin.TestIssue22295 [ABORT] (log) 
=== RUN   TestIssue22295
( GOPATH=$TMPDIR PWD=$TMPDIR/src/testplugin LD_LIBRARY_PATH=$TMPDIR/src/testplugin /opt/s/w/ir/x/w/goroot/bin/go build -gcflags '' -buildmode=plugin -o issue.22295.so ./issue22295.pkg )
( GOPATH=$TMPDIR PWD=$TMPDIR/src/testplugin LD_LIBRARY_PATH=$TMPDIR/src/testplugin /opt/s/w/ir/x/w/goroot/bin/go build -gcflags '' -o issue22295.exe ./issue22295.pkg/main.go )
SIGQUIT: quit
PC=0x7fff20766cbe m=0 sigcode=0

goroutine 0 gp=0x8cf4a00 m=0 mp=0x8cf5200 [idle]:
runtime.pthread_cond_wait(0x8cf5818, 0x8cf57d8)
	/opt/s/w/ir/x/w/goroot/src/runtime/sys_darwin.go:507 +0x2e fp=0x7ffee818bb90 sp=0x7ffee818bb68 pc=0x8accece
runtime.semasleep(0xffffffffffffffff)
...
r12    0x8cf57d8
r13    0x16
r14    0x460100004700
r15    0x4600
rip    0x7fff20766cbe
rflags 0x247
cs     0x7
fs     0x0
gs     0x0
*** Test killed with quit: ran too long (10m0s).
2024-03-29 16:41 gotip-darwin-amd64_11 go@8f59b28f cmd/compile/internal/loopvar.TestLoopVarHashes [ABORT] (log) 
=== RUN   TestLoopVarHashes
    exec.go:193: test timed out while running command: /opt/s/w/ir/x/w/goroot/bin/go run -trimpath cmd/compile/internal/loopvar/testdata/inlines
    loopvar_test.go:203: SIGQUIT: quit
        PC=0x7fff20766cbe m=0 sigcode=0
        
        goroutine 0 gp=0x7fd6c40 m=0 mp=0x7fd8940 [idle]:
        runtime.pthread_cond_wait(0x7fd8f58, 0x7fd8f18)
        	runtime/sys_darwin.go:507 +0x2e fp=0x7ffee97a8a20 sp=0x7ffee97a89f8 pc=0x74b412e
        runtime.semasleep(0xffffffffffffffff)
        	runtime/os_darwin.go:71 +0xaf fp=0x7ffee97a8a78 sp=0x7ffee97a8a20 pc=0x748f58f
...
panic: test timed out after 9m0s
running tests:
	TestLoopVarHashes (4m51s)

goroutine 119 gp=0xc000084fc0 m=0 mp=0xfbbe800 [running]:
panic({0xf9bbce0?, 0xc0000a87e0?})
	/opt/s/w/ir/x/w/goroot/src/runtime/panic.go:778 +0x168 fp=0xc000217f10 sp=0xc000217e60 pc=0xf75ca68
testing.(*M).startAlarm.func1()
	/opt/s/w/ir/x/w/goroot/src/testing/testing.go:2366 +0x385 fp=0xc000217fe0 sp=0xc000217f10 pc=0xf83e3c5
runtime.goexit({})
...
runtime.gopark(0x1e55a1518b9?, 0x3?, 0xda?, 0x10?, 0xc0001d2fb8?)
	/opt/s/w/ir/x/w/goroot/src/runtime/proc.go:402 +0xce fp=0xc0001d2f38 sp=0xc0001d2f18 pc=0xf76036e
runtime.gcBgMarkWorker(0xc000090620)
	/opt/s/w/ir/x/w/goroot/src/runtime/mgc.go:1333 +0xe9 fp=0xc0001d2fc8 sp=0xc0001d2f38 pc=0xf741189
runtime.gcBgMarkStartWorkers.gowrap1()
	/opt/s/w/ir/x/w/goroot/src/runtime/mgc.go:1249 +0x25 fp=0xc0001d2fe0 sp=0xc0001d2fc8 pc=0xf741065
runtime.goexit({})
	/opt/s/w/ir/x/w/goroot/src/runtime/asm_amd64.s:1695 +0x1 fp=0xc0001d2fe8 sp=0xc0001d2fe0 pc=0xf797ee1
created by runtime.gcBgMarkStartWorkers in goroutine 99
	/opt/s/w/ir/x/w/goroot/src/runtime/mgc.go:1249 +0x105
2024-03-29 16:41 gotip-darwin-amd64_11 go@8f59b28f cmd/compile/internal/ssa.TestDebugLines_53456 (log) 
=== RUN   TestDebugLines_53456
Preserving temporary directory /opt/s/w/ir/x/t/debug_lines_test563599907
About to run (cd /opt/s/w/ir/x/t/debug_lines_test563599907; BOTO_CONFIG=/opt/s/w/ir/x/a/gsutil-bbagent/.boto BUILDBUCKET_EXPERIMENTAL=FALSE CIPD_ARCHITECTURE=amd64 CIPD_CACHE_DIR=/opt/s/w/ir/cache/cipd_cache CIPD_PROTOCOL=v2 CLOUDSDK_CONFIG=/opt/s/w/ir/x/a/gcloud-bbagent DOCKER_CONFIG=/opt/s/w/ir/x/a/docker-cfg-bbagent DOCKER_TMPDIR=/opt/s/w/ir/x/a/docker-tmp-bbagent FIREBASE_TOKEN=ignored-non-empty-value FIREBASE_TOKEN_URL=http://127.0.0.1:49271 GCE_METADATA_HOST=127.0.0.1:49270 GCE_METADATA_IP=127.0.0.1:49270 GCE_METADATA_ROOT=127.0.0.1:49270 GIT_TERMINAL_PROMPT=0 GOARCH=amd64 GOBIN=/opt/s/w/ir/x/w/goroot/bin GOCACHE=/opt/s/w/ir/x/w/gocache GOHOSTARCH=amd64 GOHOSTOS=darwin GOOS=linux GOPATH=/opt/s/w/ir/x/w/gopath GOPLSCACHE=/opt/s/w/ir/x/w/goplscache GOROOT=/opt/s/w/ir/x/w/goroot GOROOT_BOOTSTRAP=/opt/s/w/ir/cache/tools/go_bootstrap GOTOOLCHAIN=local GO_BUILDER_NAME=gotip-darwin-amd64_11 INFRA_GIT_WRAPPER_HOME=/opt/s/w/ir/x/a/git-home-bbagent ISOLATED_RESOLVED_PACKAGE_VERSIONS_FILE=/opt/s/w/itt2s7mrit/cipd_info.json LOGDOG_COORDINATOR_HOST=logs.chromium.org LOGDOG_NAMESPACE=u/step/10/u LOGDOG_STREAM_PREFIX=buildbucket/cr-buildbucket/8752109904023604161 LOGDOG_STREAM_PROJECT=golang LOGDOG_STREAM_SERVER_PATH=unix:/opt/s/w/ir/x/ld/sock.2032863246 LOGNAME=chrome-bot LUCI_CONTEXT=/opt/s/w/ir/x/t/luci_context.1140294156 MAC_CHROMIUM_TMPDIR=/opt/s/w/ir/x/t NO_GCE_CHECK=False PYTHONCOERCECLOCALE=0 PYTHONNOUSERSITE=1 SHIVAS_BINARY=/opt/shivas/shivas SHIVAS_NAMESPACE=browser SHIVAS_RO_CRED=/opt/creds/service_accounts/service-account-machine-db-ro.json SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.3MWTVVVI0T/Listeners SWARMING_BOT_ID=mac-328-h526 SWARMING_EXTERNAL_BOT_SETUP=1 SWARMING_HEADLESS=1 SWARMING_SERVER=https://chromium-swarm.appspot.com SWARMING_TASK_ID=68a426a72255b211 TEMP=/opt/s/w/ir/x/t TEMPDIR=/opt/s/w/ir/x/t TMP=/opt/s/w/ir/x/t TMPDIR=/opt/s/w/ir/x/t VPYTHON_VIRTUALENV_ROOT=/opt/s/w/ir/cache/vpython XPC_FLAGS=0x0 XPC_SERVICE_NAME=org.swarm.bot __CF_USER_TEXT_ENCODING=0x1F6:0x0:0x0 GOENV=off GOMODCACHE=/opt/s/w/ir/x/w/gopath/pkg/mod GOPROXY=https://proxy.golang.org,direct GOSUMDB=sum.golang.org GOTOOLDIR=/opt/s/w/ir/x/w/goroot/pkg/tool/darwin_amd64 'GOVERSION=devel 8f59b28fb3b4539b182296a0e5cda51aa9df9e0c' GCCGO=gccgo GOAMD64=v1 AR=ar CC=clang CXX=clang++ CGO_ENABLED=1 TERM=dumb GO386=sse2 GOARM=7 GOARM64=v8.0 GOMIPS=hardfloat GOMIPS64=hardfloat GOPPC64=power8 GORISCV64=rva20u64 LANG=C LANGUAGE=en_US.UTF8 GOWORK=off GOTRACEBACK=system PWD=/opt/s/w/ir/x/w/goroot/src/cmd/compile/internal/ssa GOSSADIR=/opt/s/w/ir/x/t/debug_lines_test563599907 /opt/s/w/ir/x/w/goroot/bin/go build -o foo.o '-gcflags=-d=ssa/genssa/dump=(*T).Inc -N -l' /opt/s/w/ir/x/w/goroot/src/cmd/compile/internal/ssa/testdata/b53456.go )
    exec.go:193: test timed out while running command: /opt/s/w/ir/x/w/goroot/bin/go build -o foo.o -gcflags=-d=ssa/genssa/dump=(*T).Inc -N -l /opt/s/w/ir/x/w/goroot/src/cmd/compile/internal/ssa/testdata/b53456.go
    debug_lines_test.go:162: error running cmd (cd /opt/s/w/ir/x/t/debug_lines_test563599907; BOTO_CONFIG=/opt/s/w/ir/x/a/gsutil-bbagent/.boto BUILDBUCKET_EXPERIMENTAL=FALSE CIPD_ARCHITECTURE=amd64 CIPD_CACHE_DIR=/opt/s/w/ir/cache/cipd_cache CIPD_PROTOCOL=v2 CLOUDSDK_CONFIG=/opt/s/w/ir/x/a/gcloud-bbagent DOCKER_CONFIG=/opt/s/w/ir/x/a/docker-cfg-bbagent DOCKER_TMPDIR=/opt/s/w/ir/x/a/docker-tmp-bbagent FIREBASE_TOKEN=ignored-non-empty-value FIREBASE_TOKEN_URL=http://127.0.0.1:49271 GCE_METADATA_HOST=127.0.0.1:49270 GCE_METADATA_IP=127.0.0.1:49270 GCE_METADATA_ROOT=127.0.0.1:49270 GIT_TERMINAL_PROMPT=0 GOARCH=amd64 GOBIN=/opt/s/w/ir/x/w/goroot/bin GOCACHE=/opt/s/w/ir/x/w/gocache GOHOSTARCH=amd64 GOHOSTOS=darwin GOOS=linux GOPATH=/opt/s/w/ir/x/w/gopath GOPLSCACHE=/opt/s/w/ir/x/w/goplscache GOROOT=/opt/s/w/ir/x/w/goroot GOROOT_BOOTSTRAP=/opt/s/w/ir/cache/tools/go_bootstrap GOTOOLCHAIN=local GO_BUILDER_NAME=gotip-darwin-amd64_11 INFRA_GIT_WRAPPER_HOME=/opt/s/w/ir/x/a/git-home-bbagent ISOLATED_RESOLVED_PACKAGE_VERSIONS_FILE=/opt/s/w/itt2s7mrit/cipd_info.json LOGDOG_COORDINATOR_HOST=logs.chromium.org LOGDOG_NAMESPACE=u/step/10/u LOGDOG_STREAM_PREFIX=buildbucket/cr-buildbucket/8752109904023604161 LOGDOG_STREAM_PROJECT=golang LOGDOG_STREAM_SERVER_PATH=unix:/opt/s/w/ir/x/ld/sock.2032863246 LOGNAME=chrome-bot LUCI_CONTEXT=/opt/s/w/ir/x/t/luci_context.1140294156 MAC_CHROMIUM_TMPDIR=/opt/s/w/ir/x/t NO_GCE_CHECK=False PYTHONCOERCECLOCALE=0 PYTHONNOUSERSITE=1 SHIVAS_BINARY=/opt/shivas/shivas SHIVAS_NAMESPACE=browser SHIVAS_RO_CRED=/opt/creds/service_accounts/service-account-machine-db-ro.json SSH_AUTH_SOCK=/private/tmp/com.apple.launchd.3MWTVVVI0T/Listeners SWARMING_BOT_ID=mac-328-h526 SWARMING_EXTERNAL_BOT_SETUP=1 SWARMING_HEADLESS=1 SWARMING_SERVER=https://chromium-swarm.appspot.com SWARMING_TASK_ID=68a426a72255b211 TEMP=/opt/s/w/ir/x/t TEMPDIR=/opt/s/w/ir/x/t TMP=/opt/s/w/ir/x/t TMPDIR=/opt/s/w/ir/x/t VPYTHON_VIRTUALENV_ROOT=/opt/s/w/ir/cache/vpython XPC_FLAGS=0x0 XPC_SERVICE_NAME=org.swarm.bot __CF_USER_TEXT_ENCODING=0x1F6:0x0:0x0 GOENV=off GOMODCACHE=/opt/s/w/ir/x/w/gopath/pkg/mod GOPROXY=https://proxy.golang.org,direct GOSUMDB=sum.golang.org GOTOOLDIR=/opt/s/w/ir/x/w/goroot/pkg/tool/darwin_amd64 'GOVERSION=devel 8f59b28fb3b4539b182296a0e5cda51aa9df9e0c' GCCGO=gccgo GOAMD64=v1 AR=ar CC=clang CXX=clang++ CGO_ENABLED=1 TERM=dumb GO386=sse2 GOARM=7 GOARM64=v8.0 GOMIPS=hardfloat GOMIPS64=hardfloat GOPPC64=power8 GORISCV64=rva20u64 LANG=C LANGUAGE=en_US.UTF8 GOWORK=off GOTRACEBACK=system PWD=/opt/s/w/ir/x/w/goroot/src/cmd/compile/internal/ssa GOSSADIR=/opt/s/w/ir/x/t/debug_lines_test563599907 /opt/s/w/ir/x/w/goroot/bin/go build -o foo.o '-gcflags=-d=ssa/genssa/dump=(*T).Inc -N -l' /opt/s/w/ir/x/w/goroot/src/cmd/compile/internal/ssa/testdata/b53456.go ): exit status 2
        stdout:
        stderr:
        SIGQUIT: quit
        PC=0x7fff20766cbe m=0 sigcode=0
        
...
        r13    0x16
        r14    0x320100003300
        r15    0x3100
        rip    0x7fff20766cbe
        rflags 0x247
        cs     0x7
        fs     0x0
        gs     0x0
        
--- FAIL: TestDebugLines_53456 (123.21s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-04-19 17:39 gotip-darwin-amd64-longtest go@c260de24 cmd/go.TestScript/cover_pkgall_runtime (log) 
=== RUN   TestScript/cover_pkgall_runtime
=== PAUSE TestScript/cover_pkgall_runtime
=== CONT  TestScript/cover_pkgall_runtime
    script_test.go:136: 2024-04-19T22:03:04Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-134986372/tmpdir362963032/cover_pkgall_runtime1194633087
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-134986372/tmpdir362963032/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x0
        r14    0xc000105880
        r15    0xffda3ffef
        rip    0x7ff805b0ba60
        rflags 0x246
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:160: FAIL: testdata/script/cover_pkgall_runtime.txt:12: go test -coverpkg=all -race x: exit status 2
--- FAIL: TestScript/cover_pkgall_runtime (103.59s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-04-26 13:32 gotip-darwin-amd64-longtest go@4370cfbd cmd/go.TestScript/vet_flags (log) 
=== RUN   TestScript/vet_flags
=== PAUSE TestScript/vet_flags
=== CONT  TestScript/vet_flags
    script_test.go:136: 2024-04-26T18:14:51Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-3950273633/tmpdir1871686850/vet_flags2222620532
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-3950273633/tmpdir1871686850/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r14    0xc000474700
        r15    0x1ffff
        rip    0x7ff81a835a60
        rflags 0x246
        cs     0x7
        fs     0x0
        gs     0x0
        [exit status 2]
    script_test.go:160: FAIL: testdata/script/vet_flags.txt:43: context deadline exceeded
--- FAIL: TestScript/vet_flags (44.53s)
2024-04-26 18:12 gotip-darwin-amd64-longtest go@b384ee7c cmd/go.TestScript/test_cache_inputs (log) 
=== RUN   TestScript/test_cache_inputs
=== PAUSE TestScript/test_cache_inputs
=== CONT  TestScript/test_cache_inputs
    script_test.go:136: 2024-04-26T19:52:40Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1058833139/tmpdir1217724554/test_cache_inputs2584980790
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1058833139/tmpdir1217724554/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x2c0100002d00
        r14    0x7ff84d802e80
        r15    0x2c00
        rip    0x7ff80a1ed5d6
        rflags 0x247
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:160: FAIL: testdata/script/test_cache_inputs.txt:97: go test testcache -run=DirList: exit status 2
--- FAIL: TestScript/test_cache_inputs (85.27s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-04-27 11:30 gotip-darwin-amd64-longtest go@774d5b36 cmd/go.TestScript/cgo_suspect_flag_force_external (log) 
=== RUN   TestScript/cgo_suspect_flag_force_external
=== PAUSE TestScript/cgo_suspect_flag_force_external
=== CONT  TestScript/cgo_suspect_flag_force_external
    script_test.go:136: 2024-04-27T12:51:24Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1172273717/tmpdir3472394522/cgo_suspect_flag_force_external2821333579
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-1172273717/tmpdir3472394522/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        cs     0x7
        fs     0x0
        gs     0x0
        [exit status 2]
    script_test.go:160: FAIL: testdata/script/cgo_suspect_flag_force_external.txt:127: wait: testdata/script/cgo_suspect_flag_force_external.txt:105: go build -ldflags=-tmpdir=tmp2 -o /dev/null ./usesInternalCgo: exit status 2
        testdata/script/cgo_suspect_flag_force_external.txt:112: go build -ldflags=-tmpdir=tmp3 -o /dev/null ./usesInternalCgo: exit status 2
        testdata/script/cgo_suspect_flag_force_external.txt:117: go build -ldflags=-tmpdir=tmp4 -o /dev/null ./usesExplicitCgo: exit status 2
        testdata/script/cgo_suspect_flag_force_external.txt:122: go build '-ldflags=-tmpdir=tmp5 -linkmode=internal' -o /dev/null ./usesInternalCgo: exit status 2
        testdata/script/cgo_suspect_flag_force_external.txt:125: go build -o p.a ./nonMainPackageUsesExplicitCgo: exit status 2
--- FAIL: TestScript/cgo_suspect_flag_force_external (56.06s)

— watchflakes

@gopherbot
Copy link

Found new dashboard test flakes for:

#!watchflakes
post <- goos == "darwin" && log ~ `os/exec\.\(\*Cmd\)\.awaitGoroutines` && log ~ `internal/poll\.\(\*pollDesc\)\.waitRead` &&
	!(builder ~ `(gotip|go1\.\d\d)-` && date < "2024-01-18")
2024-04-30 19:45 gotip-darwin-amd64-longtest go@959e65c4 cmd/go.TestScript/test_rebuildall (log) 
=== RUN   TestScript/test_rebuildall
=== PAUSE TestScript/test_rebuildall
=== CONT  TestScript/test_rebuildall
    script_test.go:136: 2024-04-30T21:45:02Z
    script_test.go:138: $WORK=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-2900302542/tmpdir3648406598/test_rebuildall2961717021
    script_test.go:160: 
        PATH=/Users/swarming/.swarming/w/ir/x/t/cmd-go-test-2900302542/tmpdir3648406598/testbin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/x/w/goroot/bin:/Users/swarming/.swarming/w/ir/cache/tools/bin:/Users/swarming/.swarming/w/ir/bbagent_utility_packages:/Users/swarming/.swarming/w/ir/bbagent_utility_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages:/Users/swarming/.swarming/w/ir/cipd_bin_packages/bin:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3:/Users/swarming/.swarming/w/ir/cipd_bin_packages/cpython3/bin:/Users/swarming/.swarming/w/ir/cache/cipd_client:/Users/swarming/.swarming/w/ir/cache/cipd_client/bin:/Users/swarming/.swarming/cipd_cache/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/sbin:/usr/sbin:/sbin
        HOME=/no-home
        CCACHE_DISABLE=1
        GOARCH=amd64
...
        r13    0x0
        r14    0xc0000096c0
        r15    0x1fffffffffefc
        rip    0x7ff803e66a60
        rflags 0x246
        cs     0x7
        fs     0x0
        gs     0x0
    script_test.go:160: FAIL: testdata/script/test_rebuildall.txt:8: go test -x -a -c testdata/dep_test.go: exit status 2
--- FAIL: TestScript/test_rebuildall (41.11s)

— watchflakes

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Darwin
Projects
Go Compiler / Runtime
Status: Todo
Test Flakes
Status: No status
Milestone
Go1.23
Development

No branches or pull requests
5 participants
@mknyszek @dmitshur @bcmills @panjf2000 @gopherbot