Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/internal/socket: TestRace is flaky #37338

Closed
bcmills opened this issue Feb 20, 2020 · 17 comments
Closed

x/net/internal/socket: TestRace is flaky #37338

bcmills opened this issue Feb 20, 2020 · 17 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. RaceDetector release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 20, 2020

An example from the linux-amd64-fedora builder: https://build.golang.org/log/3d48fc241a76a29308028aa219a2114b4c0a5b7b

--- FAIL: TestRace (18.25s)
    --- FAIL: TestRace/test_0 (16.84s)
        socket_test.go:365: race not detected for test 0: err:<nil> out:
FAIL
FAIL	golang.org/x/net/internal/socket	18.264s

I have seen this failure many other times as well, but I can't easily collect them due to #35515.

CC @randall77 @odeke-em

@bcmills bcmills added Testing An issue that has been verified to require only test changes, not just a test failure. RaceDetector NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Feb 20, 2020
@gopherbot gopherbot added this to the Unreleased milestone Feb 20, 2020
@bcmills
Copy link
Contributor Author

bcmills commented Feb 20, 2020

See previously #35668, #35547.

@bcmills
Copy link
Contributor Author

bcmills commented Nov 6, 2020

@bcmills
Copy link
Contributor Author

bcmills commented Nov 6, 2020

2020-10-31T05:49:03-ff519b6/linux-amd64-fedora
2020-10-31T05:49:03-ff519b6/linux-amd64-race
2020-10-31T05:49:03-ff519b6/linux-ppc64le-buildlet
2020-10-31T05:49:03-ff519b6/linux-ppc64le-power9osu
2020-10-29T22:17:08-28c70e6/linux-amd64-race
2020-10-29T05:33:32-8adf50f/linux-amd64-race
2020-10-29T05:33:32-8adf50f/linux-ppc64le-buildlet
2020-10-29T05:33:32-8adf50f/linux-ppc64le-power9osu
2020-10-29T03:25:33-e049550/linux-amd64-race
2020-10-29T03:25:33-e049550/linux-ppc64le-power9osu
2020-10-27T13:37:19-8eef523/linux-amd64-fedora
2020-10-27T13:37:19-8eef523/linux-amd64-race
2020-10-26T09:15:29-146b70c/linux-amd64
2020-10-26T09:15:29-146b70c/linux-ppc64le-buildlet
2020-10-26T09:15:29-146b70c/linux-ppc64le-power9osu
2020-10-24T04:28:10-be3efd7/linux-amd64-race
2020-10-24T04:28:10-be3efd7/linux-ppc64le-buildlet
2020-10-22T23:12:55-08b3837/linux-amd64-race
2020-10-22T23:12:55-08b3837/linux-ppc64le-buildlet
2020-10-21T03:54:29-f585440/linux-amd64-clang
2020-10-21T03:54:29-f585440/linux-amd64-fedora
2020-10-21T03:54:29-f585440/linux-amd64-race
2020-10-20T06:53:57-d65d470/linux-amd64
2020-10-20T06:53:57-d65d470/linux-amd64-clang
2020-10-20T06:53:57-d65d470/linux-amd64-fedora
2020-10-20T06:53:57-d65d470/linux-amd64-race
2020-10-20T06:53:57-d65d470/linux-ppc64le-buildlet
2020-10-20T06:53:57-d65d470/linux-ppc64le-power9osu
2020-10-16T16:51:38-7b1cca2/linux-amd64-race
2020-10-16T16:51:38-7b1cca2/linux-ppc64le-buildlet
2020-10-16T16:51:38-7b1cca2/linux-ppc64le-power9osu
2020-10-10T22:47:23-4f7140c/linux-amd64-race
2020-10-10T22:47:23-4f7140c/linux-ppc64le-power9osu
2020-10-09T03:24:41-dbdefad/linux-amd64-race
2020-10-09T03:24:41-dbdefad/linux-ppc64le-buildlet
2020-10-09T03:24:41-dbdefad/linux-ppc64le-power9osu
2020-10-08T22:28:04-59f7323/linux-amd64-race
2020-10-08T22:28:04-59f7323/linux-ppc64le-power9osu
2020-10-06T15:34:59-a7d1128/linux-amd64-race
2020-10-06T15:34:59-a7d1128/linux-amd64-sid
2020-10-06T15:34:59-a7d1128/linux-ppc64le-power9osu
2020-10-02T20:24:02-0a1ea39/linux-amd64-race
2020-10-02T20:24:02-0a1ea39/linux-ppc64le-buildlet
2020-02-22T12:55:58-5a598a2/linux-amd64-clang
2020-02-22T03:33:25-078779b/linux-amd64-fedora
2020-02-22T03:33:25-078779b/linux-ppc64le-power9osu
2020-02-19T18:36:55-4628272/linux-amd64-fedora
2020-02-19T18:36:38-f229aea/linux-amd64-fedora
2020-02-19T18:36:38-f229aea/linux-amd64-race
2020-02-19T18:36:38-f229aea/linux-ppc64le-buildlet
2020-02-19T18:36:38-f229aea/linux-ppc64le-power9osu
2020-02-02T09:46:26-1617124/linux-amd64-fedora
2020-02-02T09:46:26-1617124/linux-amd64-race
2020-02-02T09:46:26-1617124/linux-amd64-sid
2020-02-02T09:46:26-1617124/linux-ppc64le-power9osu
2020-01-14T15:54:13-6afb519/linux-amd64-jessie
2020-01-14T15:54:13-6afb519/linux-amd64-race
2020-01-14T15:54:13-6afb519/linux-ppc64le-power9osu
2019-12-09T16:08:50-c0dbc17/linux-amd64-clang
2019-12-09T16:08:50-c0dbc17/linux-amd64-fedora
2019-12-09T16:08:50-c0dbc17/linux-amd64-jessie
2019-12-09T16:08:50-c0dbc17/linux-amd64-race
2019-12-09T16:08:50-c0dbc17/linux-amd64-sid
2019-12-09T16:08:50-c0dbc17/linux-ppc64le-buildlet
2019-12-09T16:08:50-c0dbc17/linux-ppc64le-power9osu
2019-12-07T00:06:13-e7e4b65/linux-amd64-race
2019-12-07T00:06:13-e7e4b65/linux-ppc64le-buildlet
2019-12-07T00:06:13-e7e4b65/linux-ppc64le-power9osu
2019-12-07T00:05:07-51f0931/linux-amd64-fedora
2019-12-07T00:05:07-51f0931/linux-amd64-race
2019-12-07T00:05:07-51f0931/linux-ppc64le-buildlet
2019-12-07T00:05:07-51f0931/linux-ppc64le-power9osu
2019-12-06T10:30:17-1ddd1de/linux-amd64-race
2019-12-06T10:30:17-1ddd1de/linux-ppc64le-buildlet
2019-12-06T10:30:17-1ddd1de/linux-ppc64le-power9osu
2019-12-06T10:28:45-afd1edf/linux-ppc64le-buildlet
2019-12-06T10:28:45-afd1edf/linux-ppc64le-power9osu
2019-12-04T02:50:24-5ee1b9f/linux-ppc64le-buildlet
2019-12-04T02:50:24-5ee1b9f/linux-ppc64le-power9osu
2019-11-26T23:54:20-ef20fe5/linux-ppc64le-buildlet
2019-11-26T23:54:20-ef20fe5/linux-ppc64le-power9osu
2019-11-25T08:49:36-ffdde10/linux-amd64-race
2019-11-25T08:49:36-ffdde10/linux-ppc64le-power9osu
2019-11-24T23:54:46-72fef5d/linux-amd64-race
2019-11-24T23:54:46-72fef5d/linux-ppc64le-buildlet
2019-11-24T23:54:46-72fef5d/linux-ppc64le-power9osu
2019-11-24T23:31:50-8f7fa26/linux-amd64-jessie
2019-11-24T23:31:50-8f7fa26/linux-amd64-race
2019-11-24T23:31:50-8f7fa26/linux-ppc64le-buildlet
2019-11-24T23:31:50-8f7fa26/linux-ppc64le-power9osu
2019-11-24T23:23:54-b954d4e/linux-amd64-race
2019-11-24T23:23:54-b954d4e/linux-ppc64le-buildlet
2019-11-24T23:23:54-b954d4e/linux-ppc64le-power9osu
2019-11-24T23:18:49-4163a4d/linux-amd64-race
2019-11-24T23:18:49-4163a4d/linux-amd64-sid
2019-11-24T23:18:49-4163a4d/linux-ppc64le-buildlet
2019-11-24T23:18:49-4163a4d/linux-ppc64le-power9osu
2019-11-19T07:31:36-fc4aabc/linux-amd64
2019-11-19T07:31:36-fc4aabc/linux-amd64-race
2019-11-19T07:31:36-fc4aabc/linux-ppc64le-buildlet
2019-11-19T07:31:36-fc4aabc/linux-ppc64le-power9osu
2019-11-18T18:34:10-d06c31c/linux-ppc64le-buildlet
2019-11-18T18:34:10-d06c31c/linux-ppc64le-power9osu
2019-11-16T16:09:21-f9c8255/linux-amd64-nocgo
2019-11-16T16:09:21-f9c8255/linux-ppc64le-buildlet
2019-11-16T16:09:21-f9c8255/linux-ppc64le-power9osu
2019-11-12T18:23:07-2180aed/linux-amd64-nocgo
2019-11-12T18:23:07-2180aed/linux-ppc64le-buildlet
2019-11-12T18:23:07-2180aed/linux-ppc64le-power9osu

@bcmills
Copy link
Contributor Author

bcmills commented Nov 9, 2020

@bcmills
Copy link
Contributor Author

bcmills commented Feb 17, 2021

2021-02-05T22:35:11-b54cd94/windows-amd64-race
2021-01-19T19:43:25-5f4716e/linux-amd64
2021-01-19T19:43:25-5f4716e/linux-amd64-buster
2021-01-19T19:43:25-5f4716e/linux-amd64-fedora
2021-01-19T19:43:25-5f4716e/linux-amd64-jessie
2021-01-19T19:43:25-5f4716e/linux-amd64-race
2021-01-19T19:43:25-5f4716e/linux-amd64-sid
2021-01-19T19:43:25-5f4716e/linux-ppc64le-buildlet
2021-01-19T19:43:25-5f4716e/linux-ppc64le-power9osu
2020-12-24T01:40:10-6772e93/linux-ppc64le-buildlet
2020-12-24T01:40:10-6772e93/linux-ppc64le-power9osu
2020-12-16T05:46:12-986b41b/linux-amd64-fedora
2020-12-16T05:46:12-986b41b/linux-amd64-race
2020-12-16T05:46:12-986b41b/linux-amd64-wsl
2020-12-16T05:46:12-986b41b/linux-ppc64le-buildlet
2020-12-16T05:46:12-986b41b/linux-ppc64le-power9osu
2020-12-09T12:38:23-ac852fb/linux-amd64-fedora
2020-12-09T12:38:23-ac852fb/linux-amd64-jessie
2020-12-09T12:38:23-ac852fb/linux-amd64-race
2020-12-09T12:38:23-ac852fb/linux-ppc64le-power9osu
2020-12-07T22:46:15-747e238/linux-amd64-wsl
2020-12-07T22:46:15-747e238/linux-ppc64le-buildlet
2020-12-07T22:46:15-747e238/linux-ppc64le-power9osu
2020-12-04T03:10:36-cc386bd/windows-amd64-race
2020-12-02T16:19:06-c7110b5/linux-amd64-fedora
2020-12-02T16:19:06-c7110b5/linux-amd64-race
2020-12-02T16:19:06-c7110b5/linux-amd64-wsl
2020-12-02T16:19:06-c7110b5/linux-ppc64le-buildlet
2020-12-01T19:55:09-5d6afe9/linux-amd64-race
2020-11-20T15:19:09-8bbd829/windows-amd64-race
2020-11-10T03:11:24-69a7880/linux-amd64-clang
2020-11-10T03:11:24-69a7880/linux-amd64-fedora
2020-11-10T03:11:24-69a7880/linux-amd64-race
2020-11-10T03:11:24-69a7880/linux-amd64-sid
2020-11-10T03:11:24-69a7880/linux-ppc64le-buildlet
2020-11-10T03:11:24-69a7880/linux-ppc64le-power9osu
2020-11-09T17:26:40-a11eb1b/linux-amd64-race
2020-11-09T17:26:40-a11eb1b/linux-ppc64le-buildlet

@bcmills
Copy link
Contributor Author

bcmills commented Feb 17, 2021

This test flakes quite frequently; it really ought to be fixed.

Marking as release-blocker for Go 1.17 via #11811.

@bcmills bcmills modified the milestones: Unreleased, Go1.17 Feb 17, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Feb 22, 2021

@tklauser
Copy link
Member

@tklauser
Copy link
Member

This keeps flaking on almost every TryBot run for golang.org/x/net (see e.g. https://golang.org/cl/303909 for a recent example). Should we temporarily skip the test?

@odeke-em
Copy link
Member

odeke-em commented Mar 23, 2021 via email

@gopherbot
Copy link

Change https://golang.org/cl/303910 mentions this issue: internal/socket: skip known flaky TestRace

gopherbot pushed a commit to golang/net that referenced this issue Mar 23, 2021
This test keeps flaking on almost every TryBot run. Skip it for now.

For golang/go#37338

Change-Id: I77cdd782f3bc164ec11894683aa50c3441807ba5
Reviewed-on: https://go-review.googlesource.com/c/net/+/303910
Trust: Tobias Klauser <tobias.klauser@gmail.com>
Run-TryBot: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Emmanuel Odeke <emmanuel@orijtech.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/310269 mentions this issue: internal/socket: more logging for TestRace

gopherbot pushed a commit to golang/net that referenced this issue Apr 15, 2021
For golang/go#37338

Change-Id: I00c222ee15bce07438ac56c0dfe4770897eccbbd
Reviewed-on: https://go-review.googlesource.com/c/net/+/310269
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Tobias Klauser <tobias.klauser@gmail.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@ianlancetaylor
Copy link
Contributor

Another occurrence on a trybot. The additional logging was not helpful.

https://storage.googleapis.com/go-build-log/7e97e4e8/linux-amd64-race_f089a4dd.log

--- FAIL: TestRace (1.28s)
    socket_test.go:393: /workdir/go/bin/go version
    socket_test.go:396: go version devel 7e97e4e8ccdba9677f31ab9380802cd7613f62c5 linux/amd64
    --- FAIL: TestRace/test_1 (0.52s)
        socket_test.go:407: /workdir/go/bin/go run -race /workdir/tmp/testrace418149795/test1.go
        socket_test.go:415: race not detected for test 1: err:<nil>
FAIL

@ianlancetaylor
Copy link
Contributor

The test works by using go run -race to run files that import "golang.org/x/net/ipv4". I wonder if anything ensures that the current package sources are the ones that are seen by go run.

@ianlancetaylor
Copy link
Contributor

Never mind, that should be ensured by the go.mod file.

@ianlancetaylor
Copy link
Contributor

OK, I can recreate this about 10% of the time by invoking go run myself. It looks like the problem is that the reads and writes are of just a single byte. I suspect that the race detector doesn't try to track conflicts at a byte level. If I increase the size of the message to 100 bytes, the test consistently finds a race.

@gopherbot
Copy link

Change https://golang.org/cl/312529 mentions this issue: internal/socket: use larger buffers in race test

@golang golang locked and limited conversation to collaborators Apr 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. RaceDetector release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Development

No branches or pull requests

5 participants