Navigation Menu

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: TestCancelRequestWhenSharingConnection can cause port exhaustion, leading to frequent test failures with "Only one usage of each socket address" on windows-amd64-longtest #47016

Closed
bcmills opened this issue Jul 1, 2021 · 14 comments
Closed

net/http: TestCancelRequestWhenSharingConnection can cause port exhaustion, leading to frequent test failures with "Only one usage of each socket address" on windows-amd64-longtest #47016

bcmills opened this issue Jul 1, 2021 · 14 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Go1.17

Comments

@bcmills
Copy link
Contributor

bcmills commented Jul 1, 2021 •
edited

What version of Go are you using?

release-branch.go1.15

Does this issue reproduce with the latest release?

No.

What did you expect to see?

Tests on the current release branches (release-branch.go1.15 and release-branch.go1.16) consistently passing, especially for all first class ports.

What did you see instead?

Tests on the windows-amd64-longtest builder started failing intermittently in March, got better for a while in May, and are now failing in net/http on just about every run (https://build.golang.org/?repo=&branch=release-branch.go1.15#short):
image

The failure mode consists of many Only one usage of each socket address errors (https://build.golang.org/log/30a479e652cc31796c8d900ab0494ffff5765266), sometimes accompanied by a test deadlock or timeout (https://build.golang.org/log/3ff9e1b80ee03390ebb171468f0fb00ceb23795d).

These failures result in TryBot / SlowBot failures on the release branch for unrelated backports (https://golang.org/cl/332330). Since the builders are consistently broken, there is a substantial risk that these failures will cause some other regression on this branch to go unnoticed.

CC @neild @golang/release
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker labels Jul 1, 2021
@bcmills bcmills added this to the Go1.15.14 milestone Jul 1, 2021
@cagedmantis
Copy link
Contributor

I believe the work to try and resolve this issue has been previously tracked in #45358.

@bcmills
Copy link
Contributor Author

bcmills commented Jul 2, 2021

I believe the work to try and resolve this issue has been previously tracked in #45358.

Yeah, it's closely related.

I actually tried to find that issue before filing this one, but it didn't turn up in my search results — the problem described in the issue title seems to have been addressed, and the string in the current error message isn't mentioned in that issue at all. 😕

@dmitshur
Copy link
Contributor

This flaky failure continues to be an issue but we're not going to block Go 1.15.14 on it. Moving to next milestone.

@dmitshur dmitshur modified the milestones: Go1.15.14, Go1.15.15 Jul 12, 2021
@cagedmantis cagedmantis modified the milestones: Go1.15.15, Go1.15.16 Aug 3, 2021
@cagedmantis
Copy link
Contributor

As with the previous milestone: This flaky failure continues to be an issue but we're not going to block Go 1.15.15 on it. Moving to next milestone.

@gopherbot
Copy link

Change https://golang.org/cl/339593 mentions this issue: net/http: set socket linger time to 0 in TestCancelRequestWhenSharingConnection

@gopherbot
Copy link

Change https://golang.org/cl/339673 mentions this issue: [release-branch.go1.15] net/http: set socket linger time to 0 in TestCancelRequestWhenSharingConnection

@gopherbot
Copy link

Change https://golang.org/cl/339594 mentions this issue: net/http: speed up and deflake TestCancelRequestWhenSharingConnection

@cagedmantis cagedmantis modified the milestones: Go1.15.16, Go1.15.15 Aug 4, 2021
@cagedmantis
Copy link
Contributor

We've moved it back to the Go1.15.15 milestone since it is actively blocking the release.

gopherbot pushed a commit that referenced this issue Aug 4, 2021
@neild
net/http: speed up and deflake TestCancelRequestWhenSharingConnection
6e73886 
This test made many requests over the same connection for 10
seconds, trusting that this will exercise the request cancelation
race from #41600.

Change the test to exhibit the specific race in a targeted fashion
with only two requests.

Updates #41600.
Updates #47016.

Change-Id: If99c9b9331ff645f6bb67fe9fb79b8aab8784710
Reviewed-on: https://go-review.googlesource.com/c/go/+/339594
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
@cagedmantis
Copy link
Contributor

@neild Since we think this issue exists on master (just not as likely to show up), show we create backport issues (and cls) from this issue? We would be backporting golang.org/cl/339594.

@neild
Copy link
Contributor

neild commented Aug 4, 2021

@gopherbot please open backport issues.

This was referenced Aug 4, 2021
Closed
Closed
@gopherbot
Copy link

Backport issue(s) opened: #47534 (for 1.15), #47535 (for 1.16).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

@dmitshur dmitshur changed the title net/http: frequent test failures with "Only one usage of each socket address" on windows-amd64-longtest net/http: TestCancelRequestWhenSharingConnection can cause port exhaustion, leading to frequent test failures with "Only one usage of each socket address" on windows-amd64-longtest Aug 4, 2021
@dmitshur dmitshur modified the milestones: Go1.15.15, Go1.17 Aug 4, 2021
@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. Testing An issue that has been verified to require only test changes, not just a test failure. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Aug 4, 2021
@gopherbot
Copy link

Change https://golang.org/cl/339829 mentions this issue: [release-branch.go1.15] net/http: speed up and deflake TestCancelRequestWhenSharingConnection

@dmitshur
Copy link
Contributor

dmitshur commented Aug 4, 2021

Since there are dedicated backport issues, retargeted this issue to 1.17.

Closing as fixed by CL 339594.

@dmitshur dmitshur closed this as completed Aug 4, 2021
@gopherbot
Copy link

Change https://golang.org/cl/339830 mentions this issue: [release-branch.go1.16] net/http: speed up and deflake TestCancelRequestWhenSharingConnection

gopherbot pushed a commit that referenced this issue Aug 4, 2021
@neild @heschi
[release-branch.go1.15] net/http: speed up and deflake TestCancelRequ…
e90f402 
…estWhenSharingConnection

This test made many requests over the same connection for 10
seconds, trusting that this will exercise the request cancelation
race from #41600.

Change the test to exhibit the specific race in a targeted fashion
with only two requests.

Fixes #47534.
Updates #41600.
Updates #47016.

Change-Id: If99c9b9331ff645f6bb67fe9fb79b8aab8784710
Reviewed-on: https://go-review.googlesource.com/c/go/+/339594
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
(cherry picked from commit 6e73886)
Reviewed-on: https://go-review.googlesource.com/c/go/+/339829
gopherbot pushed a commit that referenced this issue Aug 4, 2021
@neild @heschi
[release-branch.go1.16] net/http: speed up and deflake TestCancelRequ…
a6ca6d9 
…estWhenSharingConnection

This test made many requests over the same connection for 10
seconds, trusting that this will exercise the request cancelation
race from #41600.

Change the test to exhibit the specific race in a targeted fashion
with only two requests.

Fixes #47535.
Updates #41600.
Updates #47016.

Change-Id: If99c9b9331ff645f6bb67fe9fb79b8aab8784710
Reviewed-on: https://go-review.googlesource.com/c/go/+/339594
Trust: Damien Neil <dneil@google.com>
Run-TryBot: Damien Neil <dneil@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Heschi Kreinick <heschi@google.com>
(cherry picked from commit 6e73886)
Reviewed-on: https://go-review.googlesource.com/c/go/+/339830
@ferrmin ferrmin mentioned this issue Aug 5, 2021
Merged
@golang golang locked and limited conversation to collaborators Aug 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Go1.17
Development

No branches or pull requests
5 participants
@neild @cagedmantis @dmitshur @bcmills @gopherbot