> go version
go version go1.8.1 darwin/amd64

The HTTP/2 client appears to use a large amount of memory when the user opens several serial requests to the same server, reading a small amount from each very large response body, and then closing the body.

This was discovered in GoogleCloudPlatform/gcsfuse#227. gcsfuse is a fuse file system for Google Cloud Storage. Because the file system doesn't know how much of the GCS object is going to be read, when it detects what appears to be a sequential read through a file handle, it opens an HTTP request to GCS for the entire object, consuming what the client reads sequentially and then cancelling the request by closing the response body. The user in that issue noticed that making 500 such request serially causes gcsfuse to use 2 GiB of memory, even after closing all response bodies and performing garbage collection.

While on an airplane without Wi-Fi I managed to reproduce this in a self-contained program. The program starts a server with a handler that returns 1 GiB of data. Then it repeatedly makes GET requests where the client reads 10 MiB and then closes the body. The client optionally sleeps between reads from the response body, simulating a reasonable bandwidth internet connection. Afterward the program dumps a memory profile.

You can run it like this, using a little utility I wrote to easily see peak memory usage:

> go get -u github.com/jacobsa/cputime
> go get -u golang.org/x/net/http2
> go run $GOROOT/src/crypto/tls/generate_cert.go --host=localhost
> cputime ./reproduce --slow_client=[true|false] --http2=[true|false]

Here are the max RSS results I get:

• HTTP/1, fast client: 12.62 MiB
• HTTP/1, slow client: 11.29 MiB
• HTTP/2, fast client: 19.13 MiB
• HTTP/2, slow client: 450.80 MiB (!!)

450 MiB is insane, given that only one request is in flight at a time. I see that http.http2transportDefaultStreamFlow is 4 MiB, so I would expect at most 8 MiB of buffer usage for send and receive buffers, plus a few megs of overhead from the program itself.

According to the memory profile, pretty much all of that 450 MiB is in the pipe buffer written to by the client read loop. Note that the high usage persists even if I call runtime.GC several times after making each request.

I don't think the client is doing anything against the rules according to the package documentation (but please correct me if I'm wrong). So memory usage this high seems like a bug.

(By the way, the bandwidth achieved when using HTTP/2 is also generally lower than HTTP/1, and is much more variable. Should I file an issue for the poor performance too?)