Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/net/http2: TestContentEncodingNoSniffing failures with "connection reset by peer" #46762

Closed
bcmills opened this issue Jun 15, 2021 · 7 comments
Closed

x/net/http2: TestContentEncodingNoSniffing failures with "connection reset by peer" #46762

bcmills opened this issue Jun 15, 2021 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Go1.18

Comments

@bcmills
Copy link
Contributor

bcmills commented Jun 15, 2021

2021-06-14T18:27:18-04defd4/linux-ppc64le-buildlet
2021-04-05T18:03:19-a5a99cb/linux-ppc64-buildlet
2021-03-31T06:09:03-cb1fcc7/linux-ppc64le-power9osu
2021-03-29T18:18:59-df645c7/linux-ppc64le-buildlet

CC @neild @tombergan @laboger

It's not clear to me whether this is a flaky Go test or a platform bug.
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. arch-ppc64x labels Jun 15, 2021
@bcmills bcmills added this to the Backlog milestone Jun 15, 2021
@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022 •
edited

Turns out not to be specific to PPC64, so this is looking like a flaky test or x/net/http2 bug rather than a platform bug.

--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: Failed to fetch URL: write tcp 127.0.0.1:56032->127.0.0.1:34169: write: connection reset by peer
FAIL
FAIL	golang.org/x/net/http2	18.129s

greplogs --dashboard -md -l -e 'FAIL: TestContentEncodingNoSniffing .*\n( .*\n)* .*: connection reset by peer' --since=2021-06-16

2022-01-21T00:39:55-0dd24b2-32636cd/linux-amd64-clang
2021-11-03T00:07:38-4a448f8-49a0063/linux-386-clang
2021-09-03T01:38:54-8d99171-9f69a44/linux-arm64-aws

@bcmills bcmills changed the title x/net/http2: TestContentEncodingNoSniffing failures with "connection reset by peer" on linux-ppc64le x/net/http2: TestContentEncodingNoSniffing failures with "connection reset by peer" Jan 21, 2022
@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022

These failures do not appear to be specific to a single subtest.

@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022

The subtests of this test share a single Transport, but tear down and set up a new serverTester for each subtest.

@neild, could that cause this failure mode if two of the serverTester instances happen to pick the same port?

@bcmills

This comment has been minimized.

Sign in to view
@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022

Setting -httptest.serve to explicitly reuse the same port seems to increase the failure rate substantially.

~/x/net$ go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:0 -count=10000 -timeout=1h
ok      golang.org/x/net/http2  429.798s

~/x/net$ go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:8080 -count=10000 -timeout=1h
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:55808->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:55810->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56964->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40744->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40750->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46014->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46036->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46090->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56912->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35440->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40690->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:40692->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45978->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45982->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.02s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45990->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45992->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45996->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46002->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46010->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46014->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46022->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:46028->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:47740->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35336->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35340->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35342->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35348->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.05s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45878->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.03s)
    --- FAIL: TestContentEncodingNoSniffing/no_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45908->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45914->127.0.0.1:8080: write: connection reset by peer
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:45918->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: unexpected EOF
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:47706->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.06s)
    --- FAIL: TestContentEncodingNoSniffing/phony_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56668->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.02s)
    --- FAIL: TestContentEncodingNoSniffing/zlib_content-encoding,_zlibbed (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:56672->127.0.0.1:8080: write: connection reset by peer
--- FAIL: TestContentEncodingNoSniffing (0.04s)
    --- FAIL: TestContentEncodingNoSniffing/empty_but_set_content-encoding (0.00s)
        server_test.go:4231: GET https://127.0.0.1:8080: write tcp 127.0.0.1:35158->127.0.0.1:8080: write: connection reset by peer
FAIL
FAIL    golang.org/x/net/http2  430.663s
FAIL

@gopherbot
Copy link

Change https://golang.org/cl/380154 mentions this issue: http2: in TestContentEncodingNoSniffing, do not allow the Transport to outlive the serverTester

@bcmills
Copy link
Contributor Author

bcmills commented Jan 21, 2022

Marking as release-blocker: this test empirically affects multiple first-class ports, and I've mailed what I believe to be a fix.

@bcmills bcmills modified the milestones: Backlog, Go1.18 Jan 21, 2022
@bcmills bcmills added the Testing An issue that has been verified to require only test changes, not just a test failure. label Jan 21, 2022
dteh pushed a commit to dteh/fhttp that referenced this issue Jun 22, 2022
http2: in TestContentEncodingNoSniffing, do not allow the Transport t…
cf39dc4 
…o outlive the serverTester

This test constructed a new serverTester for each subtest, but reused
a Transport across all of the tests. I don't fully understand why, but
for some reason that occasionally led to "connection reset by peer"
failures when the port was reused.

Scoping the Transport to the lifetime of the serverTester seems to
resolve the errors: I tested with

	go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:8080 -count=10000

and got lots of failures prior to this change and none after.

Fixes golang/go#46762
(I hope.)

Change-Id: I385c077c1d8627e42ce4d2db041878aacb5452fb
Reviewed-on: https://go-review.googlesource.com/c/net/+/380154
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
@rsc rsc unassigned neild and bcmills Jun 23, 2022
WeiminShang added a commit to WeiminShang/net that referenced this issue Nov 16, 2022
@WeiminShang
http2: in TestContentEncodingNoSniffing, do not allow the Transport t…
f9822fc 
…o outlive the serverTester

This test constructed a new serverTester for each subtest, but reused
a Transport across all of the tests. I don't fully understand why, but
for some reason that occasionally led to "connection reset by peer"
failures when the port was reused.

Scoping the Transport to the lifetime of the serverTester seems to
resolve the errors: I tested with

	go test ./http2 -run=TestContentEncodingNoSniffing -httptest.serve=127.0.0.1:8080 -count=10000

and got lots of failures prior to this change and none after.

Fixes golang/go#46762
(I hope.)

Change-Id: I385c077c1d8627e42ce4d2db041878aacb5452fb
Reviewed-on: https://go-review.googlesource.com/c/net/+/380154
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Damien Neil <dneil@google.com>
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-Linux release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Go1.18
Development

No branches or pull requests
3 participants
@neild @bcmills @gopherbot