Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/net/http2: internal error: attempt to send frame on a closed stream #56940

Open
joliveirinha opened this issue Nov 25, 2022 · 3 comments
Open

x/net/http2: internal error: attempt to send frame on a closed stream #56940

joliveirinha opened this issue Nov 25, 2022 · 3 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unreleased

Comments

@joliveirinha
Copy link

What version of Go are you using (go version)?

1.19.3

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 linux

What did you do?

This is not an easy issue to reproduce because it seems requiring a lot of concurrency and high load of requests.
Under such high load scenario, we observe a panic in the go routine that is serving the http2 connection.

Because this panic occurs in the path of the serve(), and there is no panic recover, when it occurs, the connection is closed.
However, if the panic occurred in the scope of the runHandler, it would have been caught and recovered.
So, it is not clear to me why this particular is not covered by a panic handler.

Looking into the specific case, it seems that what is happening is that some frames are being queued to be written, and in parallel something triggered the close of a stream.
During the loop of the serve() method, frames are dequeued to be written, which encounter a frame for a stream that was already closed, thus resulting in the panic.

What did you expect to see?

The connection just fails because the panic occurs in the execution path of the serve() which doesn't recover from panics as it occurs if it was done within the runHandler method.

The http2 connection to recover from the panic and keep the http2 connection in a stable state and reusable.

What did you see instead?

panic: internal error: attempt to send frame on a closed stream: [FrameWriteRequest stream=907097, ch=false, writer=http2.writeWindowUpdate]

goroutine 136 [running]:
golang.org/x/net/http2.(*serverConn).startFrameWrite(0xc0003f2300, {{0x1386c48, 0xc053eb8ce8}, 0xc00db48000, 0x0})
    golang.org/x/net/http2/server.go:1200 +0x439
golang.org/x/net/http2.(*serverConn).scheduleFrameWrite(0xc0003f2300)
    golang.org/x/net/http2/server.go:1323 +0x105
golang.org/x/net/http2.(*serverConn).writeFrame(0xc0003f2300, {{0x1386c48?, 0xc06bcbf5d0?}, 0x0?, 0x0?})
    golang.org/x/net/http2/server.go:1176 +0x1f3
golang.org/x/net/http2.(*serverConn).sendWindowUpdate32(0xc0003f2300, 0x0, 0xea)
    golang.org/x/net/http2/server.go:2369 +0xb8
golang.org/x/net/http2.(*serverConn).sendWindowUpdate(0xc0003f2300, 0xc00098b410?, 0xc0000dd759?)
    golang.org/x/net/http2/server.go:2353 +0x95
golang.org/x/net/http2.(*serverConn).closeStream(0xc0003f2300, 0xc00db48000, {0x137de60, 0xc00007e550})
    golang.org/x/net/http2/server.go:1591 +0x194
golang.org/x/net/http2.(*serverConn).wroteFrame(0xc0003f2300, {{}, {{0x1385398, 0xc016c23dd0}, 0xc00db48000, 0xc01c2b4360}, {0x0, 0x0}})
    golang.org/x/net/http2/server.go:1266 +0x165
golang.org/x/net/http2.(*serverConn).serve(0xc0003f2300)
    golang.org/x/net/http2/server.go:908 +0xc72
golang.org/x/net/http2.(*Server).ServeConn(0xc0007601c0, {0x1391f50?, 0xc0001ec000}, 0xc000018ef0)
    golang.org/x/net/http2/server.go:504 +0xbc5
@joliveirinha joliveirinha changed the title affected/package: net/http2 x/net: affected/package: net/http2 Nov 25, 2022
@gopherbot gopherbot added this to the Unreleased milestone Nov 25, 2022
@joliveirinha
Copy link
Author

@bradfitz not sure if this is something you can help quickly identifying the root cause.

@mdlayher mdlayher changed the title x/net: affected/package: net/http2 x/net/http2: internal error: attempt to send frame on a closed stream Nov 25, 2022
@seankhliao
Copy link
Member

and you're sure this isn't because your code has a data race?

@joliveirinha
Copy link
Author

I don't want to discard that possibility.

But I find it difficult. Mainly because:

  1. this is a very are event
  2. the panic happens on internal data structures of the net/http2 package when handling frames to write to the conn
  3. I don't see how our usage of public API (response writer and request) can trigger this
  4. Also, reading the code and the place where it is, it seems to take into consideration that panics can occur.
  5. A quick search and we can find some past issues related to this part of the code.

Because this part of the code of complex, I am trying to understand:

  1. what is the order of events that can cause a stream to be closed while there are pending frames to be written, and that afterwards causing this panic.
  2. if 1) is a valid scenario that should be covered by the internals of this package, and if so, should it handle gracefully?

@seankhliao seankhliao added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Nov 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
3 participants
@joliveirinha @gopherbot @seankhliao