Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

runtime: TestNetpollDeadlock flakes across the board #35353

Closed
FiloSottile opened this issue Nov 4, 2019 · 7 comments
Closed

runtime: TestNetpollDeadlock flakes across the board #35353

FiloSottile opened this issue Nov 4, 2019 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Milestone
Go1.14

Comments

@FiloSottile
Copy link
Contributor

FiloSottile commented Nov 4, 2019 •
edited

Saw it on a freebsd-amd64-12_0 TryBot, but it's been all over the place over the last few days.

--- FAIL: TestNetpollDeadlock (60.09s)
    crash_test.go:95: testprognet NetpollDeadlock exit status: exit status 2
    crash_test.go:450: output does not start with "done\n":
        dialing
        SIGQUIT: quit
        PC=0x45f1ae m=0 sigcode=65537

2019-11-04T17:19:36-1da575a/linux-amd64
2019-11-04T15:18:34-d3660e8/linux-amd64-clang
2019-11-04T10:13:30-1b2ff10/netbsd-amd64-8_0
2019-11-03T04:58:01-32190b0/freebsd-amd64-race
2019-11-03T04:57:50-c6621d9/freebsd-amd64-race
2019-11-03T01:44:46-d2c039f/linux-amd64-nocgo
2019-11-02T21:51:16-d16ec13/linux-ppc64le-power9osu
2019-11-02T21:51:14-a3ffb0d/freebsd-arm64-dmgk
2019-11-02T21:51:14-a3ffb0d/linux-arm64-packet
2019-11-02T19:25:54-afe50c1/linux-amd64
2019-11-02T00:36:59-48a0b97/openbsd-amd64-64
2019-11-01T22:04:05-130f3c0/darwin-amd64-10_14
2019-11-01T20:16:01-c9d89f6/darwin-amd64-race
2019-11-01T14:45:24-c0555a2/freebsd-amd64-12_0
2019-11-01T13:38:23-24e9c53/freebsd-amd64-race
2019-11-01T05:38:51-e96fd13/freebsd-386-11_2
2019-11-01T05:38:51-e96fd13/linux-amd64-race
2019-11-01T01:51:26-efd395f/freebsd-arm64-dmgk
2019-11-01T00:04:51-4a09a9b/linux-amd64-jessie
2019-10-31T20:45:40-d6f2f83/freebsd-386-12_0
2019-10-31T17:21:56-48c0cef/freebsd-386-11_2
2019-10-31T17:09:48-a9b37ae/linux-ppc64le-buildlet

/cc @ianlancetaylor
@FiloSottile FiloSottile added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Nov 4, 2019
@FiloSottile FiloSottile added this to the Go1.14 milestone Nov 4, 2019
@FiloSottile
Copy link
Contributor Author

Seems to have coincided with this rarer semi-related flake, which I noticed at https://storage.googleapis.com/go-build-log/b7ab0107/linux-amd64_99c6e062.log

runtime: netpollBreak write failed with 9
fatal error: runtime: netpollBreak write failed

2019-11-04T16:32:38-7dcd343/darwin-amd64-nocgo
2019-11-02T21:51:21-177a36a/dragonfly-amd64
2019-11-02T21:51:14-a3ffb0d/darwin-amd64-10_14
2019-11-02T21:51:07-40b7455/darwin-amd64-nocgo
2019-11-02T05:52:33-8de0bb7/netbsd-amd64-8_0
2019-11-01T21:41:41-9bde9b4/darwin-amd64-10_14
2019-11-01T05:38:51-e96fd13/darwin-amd64-10_14

@bradfitz
Copy link
Contributor

bradfitz commented Nov 4, 2019

Maybe https://go-review.googlesource.com/c/go/+/204800 fixes it.

@ianlancetaylor
Copy link
Contributor

The netpollBreak write failed with 9 failures should be fixed by https://golang.org/cl/204937, which is awaiting review.

I'm less sure about the TestNetpollDeadlock failures. Let's see if they still happen now that CL 204800 is in.

@FiloSottile
Copy link
Contributor Author

(FYI you did not assign a reviewer on CL 204937.)

@ianlancetaylor
Copy link
Contributor

It wasn't in anyone's particular speciality. I left it for the default approach.

gopherbot pushed a commit that referenced this issue Nov 5, 2019
@ianlancetaylor
runtime: don't return from netpollGenericInit until init is complete
04e80fb 
As a side-effect ensure that netpollinited only reports true when
netpoll initialization is complete.

Fixes #35282
Updates #35353

Change-Id: I21f08a04fcf229e0de5e6b5ad89c990426ae9b89
Reviewed-on: https://go-review.googlesource.com/c/go/+/204937
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/204937 mentions this issue: runtime: don't return from netpollGenericInit until init is complete

@danscales
Copy link
Contributor

I reproduced this on linux-amd64 at 1da575a (the latest test failure above) by repeating this 100 times:

go test runtime -test.run TestNetpollDeadlock -test.count=100

(usually happened within about 20 times). Using this reproduction case, I showed that it was also fixed by Ian's commit d80ab3e ("runtime: wake netpoller when dropping P, don't sleep too long in sysmon"). The failure is reproducible in commit immediate before Ian's, and not reproducible with Ian's commit.

So, I think we can close this bug as well.

@jbpratt jbpratt mentioned this issue Nov 11, 2020
Closed
@golang golang locked and limited conversation to collaborators Nov 12, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
5 participants
@bradfitz @FiloSottile @danscales @ianlancetaylor @gopherbot