I think the Client & Server use cases & APIs are different enough that they probably warrant separate bugs.

But while it's all here for now, is func paddingScheme(int) uint8 the right API? It seems one might want to choose what padding is used in an http.Handler per each ResponseWriter.Write call. But then it's intermingled with the Flush mechanics.

But you could imagine keeping our existing Write API but saying that we look at the bytes between the Write argument's length and capacity and treat them as the padding length if those bytes are filled with some sentinel value like PADPADPADPAD. (then write them out in the padded frames as zeros or random data or whatever)

It seems one might want to choose what padding is used in an http.Handler per each ResponseWriter.Write call. But then it's intermingled with the Flush mechanics.

It sounds like you're considering a finer level of granularity here, with padding per-Write instead of per-ResponseWriter. Yes, we could achieve that by adding a WriteDataPadded function on http2.responseWriter. WriteDataPadded would just need to call Flush, and then write the additional frame directly into the serverConn.

But you could imagine keeping our existing Write API but saying that we look at the bytes between the Write argument's length and capacity and treat them as the padding length if those bytes are filled with some sentinel value like PADPADPADPAD.

This seems like a more fragile approach than the proposal, but I'm OK with any approach that you think we can land. However, I think it also has the same bad interaction with Flusher.

But you could imagine keeping our existing Write API but saying that we look at the bytes between the Write argument's length and capacity and treat them as the padding length if those bytes are filled with some sentinel value like PADPADPADPAD.

That would be very complicated to implement robustly: io.Writer does not document any reads beyond the length of the slice, so callers are allowed to (for example) send one portion of a slice to a Write call while concurrently writing to the bytes just beyond its length. Naïvely snooping those bytes could introduce a data race.

Even worse, I don't think we require that the bytes beyond the length of a slice be valid in general; in some cases (e.g. #13656), we have explicitly recommended that people construct subslices with artificially large capacities in order to alias memory from outside the Go runtime. Because we do not enforce validity beyond the length of the slice anywhere else in the runtime, we have no guarantee that the memory beyond the length of the slice is even mapped: attempting to read it may cause an unhandled fault.

What do other HTTP/2 libraries or servers do? How can we evaluate whether the proposed API supports other known use cases? It doesn't seem like Go has to lead here, and if others are leading then we should make sure we are at least consciously deviating.

Good point. The proposal here is similar to node.js's PADDING_STRATEGY_CALLBACK option.

It's unclear to me how users will actually use this. The middleware layers are not going to preserve this option. Are there best practices for how to set the padding amount? If so, why not just do it in the HTTP/2 library by default? Otherwise, what are examples of applications that would control padding, and how? And is it OK for them to lose their padding if there are any middleware layers providing their own ResponseWriters?

The middleware layers are not going to preserve this option.

That depends on the webserver, but true, some of them might have to do some changes in order to make padding work. Apparently, most popular Golang web server Caddy already supports this without any changes. Caddy middleware exposes following function to the plugins:

func (h MyHandler) ServeHTTP(w http.ResponseWriter, r *http.Request) (int, error)

The underlying type of the w here is *httpserver.ResponseRecorder, which embeds *httpserver.ResponseWriterWrapper, which in turn embeds standard http.ResponseWrapper.

I added following code to my Caddy plugin in order to determine the precise type of deeply embedded w http.ResponseWriter:

      switch w1 := w.(type) {
      case *httpserver.ResponseRecorder:
             fmt.Printf("deeply embedded type of ResponseWriter: %v\n",
                    reflect.TypeOf(w1.ResponseWriterWrapper.ResponseWriter))
      default:
             fmt.Printf("unexpected type of ResponseWriter: %v\n", w1)
      }

the result is:
deeply embedded type of ResponseWriter: *http.http2responseWriter

which is a type auto-generated from http2.ResponseWriter and bundled with http package. Therefore, developer of padding plugin could simply cast w to interface with SetPaddingScheme()

However, even if this wasn’t already the case, I suspect @mholt -- author of Caddy -- would be happy to accept a reasonable PR to make it work.

Are there best practices for how to set the padding amount?

There is no silver bullet, as the http2 spec points out: “Correct application can depend on having specific knowledge of the data that is being padded”.
There are some more or less generally useful padding schemes, good one being “pad up to the multiple of 256”. However, such scheme gives away the fact that padding is used at all, and I would personally find it useful to use different padding schemes in my projects.

Otherwise, what are examples of applications that would control padding, and how?

Proxying of protocols with fixed-size payload, such as Tor cells, makes the proxied protocol immediately obvious on the wire, since the majority of packets will be of size TOR_CELL_SIZE*N + CONST_OVERHEAD. With padding up to the multiple of 256, situation isn’t much better, and it’s clear that padding is used, which is bad, since certain network actors might aspire to censor the Tor.

And is it OK for them to lose their padding if there are any middleware layers providing their own ResponseWriters?

Could be OK, could be not OK. It’s up to application developer to decide what to do if cast to interface with SetPaddingScheme() fails: could shut down with error. Either way, this won't fail silently due to explicit cast.
That’s a good point, since this cast might fail due to underlying responseWriter being http1, or that web server stopped embedding http2 responseWriter in a desired way. Regression test to ensure correct embedding would be useful for specific applications.

In general we try not to have knobs for every possible customization (think garbage collection settings in Go vs other languages). If there is a Right Thing to do with respect to padding, then let's do that. But it sounds like the jury is out on that, in which case it seems like we should just wait until the security and HTTP/2 community figures that out.

Brad suggests trying to figure out instead a general way to turn an http.ResponseWriter into an http2.Framer, which might have broader applications. But it's still unclear how that would interact with middleware. Perhaps a new proposal to figure out a way to do that would be a way forward.

But a padding-specific change seems not worthwhile right now.