Change https://go.dev/cl/421975 mentions this issue: http2: send client conn flow control bytes back immediately

@neild

HTTP/2 has both per-connection and per-stream flow control. The behavior you're seeing is because the slow stream has consumed not just its own flow-control tokens, but the connection tokens as well. To ensure that a single stream can't consume the connection tokens, you can configure the per-stream limit to something lower than the per-connection limit.

The relevant knobs are MaxUploadBufferPerConnection and MaxUploadBufferPerStream on http2.Server.

Here's your example modified to set a higher per-connection limit:
https://go.dev/play/p/6H7_E1f7yrH

H2c Server starting on :8081
high priority req response time: 0s
low priority req response time: 5s

Thanks for the reply!

That's correct, connection tokens got exhausted which stalled other streams on the same connection. While the approach you suggested definitely works, this limit is almost always easy to exhaust when the server has a high number of concurrent requests. I have an L7 proxy which forwards requests received to different destinations. When one of the destinations A is unhealthy or slow, the request bound to that dest A starts backing up in the client connection pipeline and consumes the connection flow control tokens. Since connection flow control tokens are low, other requests bound for different destinations start starving for the connection flow control tokens.

The example you shared can be updated to handle a concurrent number of streams to reproduce the issue again.

Surprisingly, we didn't face this issue while using gRPC-go server and after diving in I found gRPC-go HTTP2 server returns connection flow control tokens immediately on receiving Data frame. Source.

You're describing having only per-stream flow control, with no per-connection flow control.

If that's what you want, then you can set MaxUploadBufferPerConnection to MaxUploadBufferPerStream*MaxConcurrentStreams, which will effectively disable connection-level flow control.

I don't understand what purpose connection-level flow control is supposed to serve if tokens are returned immediately without any pushback mechanism. Perhaps I'm missing something.