Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/net/http2: TestServer_Response_.* failures with "timeout waiting for frame" #34615

Closed
bcmills opened this issue Sep 30, 2019 · 7 comments
Closed

x/net/http2: TestServer_Response_.* failures with "timeout waiting for frame" #34615

bcmills opened this issue Sep 30, 2019 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone
Go1.18

Comments

@bcmills
Copy link
Contributor

bcmills commented Sep 30, 2019

Seen in https://build.golang.org/log/ddb26b7792d0c4fc77efb48afcb95136bf8f2c9b on the netbsd-arm-bsiegert builder:

--- FAIL: TestServer_Response_ManyHeaders_With_Continuation (2.46s)
    server_test.go:486: Error while expecting a HEADERS frame: timeout waiting for frame
    server_test.go:236: Framer read log:
        2019-09-30 13:59:22.907251019 Framer 0xa9a680: read SETTINGS len=24, settings: MAX_FRAME_SIZE=1048576, MAX_CONCURRENT_STREAMS=250, MAX_HEADER_LIST_SIZE=1048896, INITIAL_WINDOW_SIZE=1048576
        2019-09-30 13:59:22.907864436 Framer 0xa9a680: read SETTINGS flags=ACK len=0
        2019-09-30 13:59:22.90802856 Framer 0xa9a680: read WINDOW_UPDATE len=4 (conn) incr=983041
    server_test.go:242: Framer write log:
        2019-09-30 13:59:22.898343519 Framer 0xa9a680: wrote SETTINGS len=0
        2019-09-30 13:59:22.907385019 Framer 0xa9a680: wrote SETTINGS flags=ACK len=0
        2019-09-30 13:59:22.908223602 Framer 0xa9a680: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=16
FAIL
FAIL	golang.org/x/net/http2	90.954s

CC @bradfitz @tombergan @bsiegert
@bcmills bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. OS-NetBSD labels Sep 30, 2019
@bcmills bcmills added this to the Go1.14 milestone Sep 30, 2019
@bcmills bcmills changed the title x/net/http2: TestServer_Response_ManyHeaders_With_Continuation flake on netbsd-arm-bsiegert builder x/net/http2: TestServer flakes with "timeout waiting for frame" Sep 30, 2019
@bcmills bcmills removed the OS-NetBSD label Sep 30, 2019
@bcmills
Copy link
Contributor Author

bcmills commented Sep 30, 2019

A similar failure in TestServerDoS_MaxHeaderListSize on the solaris-amd64-smartosbuildlet builder. Not clear to me to what extent it is related.

--- FAIL: TestServerDoS_MaxHeaderListSize (2.50s)
    server_test.go:486: Error while expecting a HEADERS frame: timeout waiting for frame
    server_test.go:236: Framer read log:
        2019-09-30 15:57:39.43298439 Framer 0xc00036e540: read SETTINGS len=24, settings: MAX_FRAME_SIZE=1048576, MAX_CONCURRENT_STREAMS=250, MAX_HEADER_LIST_SIZE=1048896, INITIAL_WINDOW_SIZE=1048576
        2019-09-30 15:57:39.433644805 Framer 0xc00036e540: read SETTINGS flags=ACK len=0
        2019-09-30 15:57:39.433715934 Framer 0xc00036e540: read WINDOW_UPDATE len=4 (conn) incr=983041
    server_test.go:242: Framer write log:
        2019-09-30 15:57:39.432324642 Framer 0xc00036e540: wrote SETTINGS len=0
        2019-09-30 15:57:39.433026381 Framer 0xc00036e540: wrote SETTINGS flags=ACK len=0
        2019-09-30 15:57:39.433786762 Framer 0xc00036e540: wrote HEADERS flags=END_STREAM stream=1 len=4065
        2019-09-30 15:57:39.486702755 Framer 0xc00036e540: wrote CONTINUATION flags=END_HEADERS stream=1 len=1048576
FAIL
FAIL	golang.org/x/net/http2	106.144s

@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@bcmills bcmills mentioned this issue Oct 31, 2019
Open
@bcmills
Copy link
Contributor Author

bcmills commented Feb 20, 2020

https://build.golang.org/log/0f86d0d8ef59cc9e79e1f7c8ef37d3c62280d4d2

--- FAIL: TestServer_Response_LargeWrite (2.35s)
    server_test.go:512: Error while expecting a DATA frame: timeout waiting for frame
…
FAIL	golang.org/x/net/http2	12.602s

@bcmills

This comment has been minimized.

Sign in to view
@bcmills bcmills changed the title x/net/http2: TestServer flakes with "timeout waiting for frame" x/net/http2: TestServer failures with "timeout waiting for frame" Feb 25, 2021
@bcmills

This comment has been minimized.

Sign in to view
@bcmills
Copy link
Contributor Author

bcmills commented Jan 11, 2022

greplogs --dashboard -md -l -e '(?ms)FAIL: TestServer[_ ].* timeout waiting for frame.*^FAIL\s+(?:golang.org/x/)net/http'

2021-12-07T21:33:49-8537929-e08d1fb/netbsd-amd64-9_0-n1 (TestServer_Response_TransferEncoding_chunked)
2021-11-10T05:08:25-ef0fda0-17980df/freebsd-arm-paulzhol (TestServer_Response_LargeWrite)
2021-09-15T23:27:54-943fd67-552410f/freebsd-arm-paulzhol (TestServer_Response_ManyHeaders_With_Continuation)
2020-11-27T22:28:11-69a7880-0252cfd/netbsd-arm-bsiegert (TestServer_Response_ManyHeaders_With_Continuation)
2020-02-15T16:40:12-1617124-6917529/freebsd-arm-paulzhol (TestServer_Response_ManyHeaders_With_Continuation)
2019-11-24T23:54:46-72fef5d-b81dd1c/freebsd-arm-paulzhol (TestServer_Response_ManyHeaders_With_Continuation)
2019-09-30T13:41:27-c5a3c61-0c07603/netbsd-arm-bsiegert (TestServer_Response_ManyHeaders_With_Continuation)

@bcmills bcmills changed the title x/net/http2: TestServer failures with "timeout waiting for frame" x/net/http2: TestServer_Response_.* failures with "timeout waiting for frame" Jan 11, 2022
@bcmills bcmills mentioned this issue Jan 11, 2022
Closed
@bcmills bcmills modified the milestones: Backlog, Go1.18 Jan 11, 2022
@bcmills bcmills self-assigned this Jan 11, 2022
@gopherbot
Copy link

Change https://golang.org/cl/377814 mentions this issue: http2: eliminate arbitrary timeouts in readFrame methods

@bcmills
Copy link
Contributor Author

bcmills commented Jan 12, 2022

Marking as release-blocker (per #11811) due to regularly-recurring failures on multiple platforms. I have what I believe to be a fix awaiting review.

@bcmills bcmills added the Testing An issue that has been verified to require only test changes, not just a test failure. label Jan 12, 2022
dteh pushed a commit to dteh/fhttp that referenced this issue Jun 22, 2022
@neild
http2: eliminate arbitrary timeouts in readFrame methods
c42bedb 
Certain slower builders empirically do not finish before these
timeouts, and if a real deadlock should occur it would be more useful
to let the test itself time out and get a goroutine dump anyway.

Fixes golang/go#50556
Fixes golang/go#34615

Change-Id: I53ea616faa34f4ccc73af8eb18b794e12271b883
Reviewed-on: https://go-review.googlesource.com/c/net/+/377814
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Trust: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
@rsc rsc unassigned bcmills Jun 23, 2022
WeiminShang added a commit to WeiminShang/net that referenced this issue Nov 16, 2022
@WeiminShang @neild
http2: eliminate arbitrary timeouts in readFrame methods
142ad58 
Certain slower builders empirically do not finish before these
timeouts, and if a real deadlock should occur it would be more useful
to let the test itself time out and get a goroutine dump anyway.

Fixes golang/go#50556
Fixes golang/go#34615

Change-Id: I53ea616faa34f4ccc73af8eb18b794e12271b883
Reviewed-on: https://go-review.googlesource.com/c/net/+/377814
Trust: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
TryBot-Result: Gopher Robot <gobot@golang.org>
Reviewed-by: Benny Siegert <bsiegert@gmail.com>
Trust: Benny Siegert <bsiegert@gmail.com>
Reviewed-by: Damien Neil <dneil@google.com>
@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. release-blocker Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Milestone
Go1.18
Development

No branches or pull requests
3 participants
@rsc @bcmills @gopherbot