cc @bradfitz

Need your help, I had already debug with many days but did not found the root cause.

/cc

I ran the reproducer locally and reproduced it.

I haven't found where the issue is but I do know we are reusing a req.Body on the http2 transport side. What I have determined is that we read the data once and then we reuse the body which has no more data.

we are reusing a req.Body on the http2 transport side

I found http2 transport entrypoint get the request which has no data, and base that I think we are reusing a req.Body on http transport but not http2 transport. I test with adding the following codes in http2 transport entrypoint and got panic:

// RoundTripOpt is like RoundTrip, but takes options.
func (t *Transport) RoundTripOpt(req *http.Request, opt RoundTripOpt) (*http.Response, error) {
	if !(req.URL.Scheme == "https" || (req.URL.Scheme == "http" && t.AllowHTTP)) {
		return nil, errors.New("http2: unsupported scheme")
	}

	// check request body has data, panic if it's no data
	if req.Body != nil && req.GetBody != nil {
		body, err := ioutil.ReadAll(req.Body)
		if err != nil {
			return nil, err
		}

		if len(body) == 0 {
			panic("http2 roundtrip get a requst which body has no data")
		}

		var ierr error
		req.Body, ierr = req.GetBody()
		if ierr != nil {
			return nil, ierr
		}
	}

I found the issue and I have a solution but I need to better understand the flow.
The failure occurs in the http transport when the altRT.RoundTrip is taken. Later on a second RoundTrip occurs, but a new request & body wasn't created.

Took a bit of time but I figured out the problem.
An http2 request is attempted via altRT.RoundTrip. The first attempt writes the body but fails because the connection gets a GOAWAY. The http2 side retries but gets a no cached connection error which bubbles up as a ErrSkipAltProtocol.
The http side takes over and tries again but this time the body is already read.

Given we never reset the body in the existing request, I am not certain how to solve this other than no returning the ErrSkipAltProtocol if we have already written the body.

(There was work in this area recently in ea1437a for #34978 and #35113 but I haven't read this bug or those ones enough to know whether they're related.)

@bradfitz It doesn't look like it. Its a corner case that has always been broken.
Retrying on the http2 side and then returning an ErrSkipAltProtocol after a failed attempt seems odd.
I just don't know how to fix the implicit constraint that a req's body is only read once given we are allowing the http side to use a request that has already been processed.
One answer would be to always make a new request if the altRT path is taken since we can't tell what if anything has been touched.

I do have a fix that returns Get "https://127.0.0.1:45927/put/0": http2: Transport received Server's graceful shutdown GOAWAY rather than the status code: 500, body: request declared a Content-Length of 5 but only wrote 0 bytes. Normally one might see http2: no cached connection was available but that would normally be replaced by the ErrSkipAltProtocol.
The other option would be to always create a new request to allow this request to be reused later if necessary.

I misunderstood that http2 request will never take altRT.RoundTrip before.

What about we fix http roundtrip altRT.RoundTrip error process like:

	if altRT := t.alternateRoundTripper(req); altRT != nil {
		if resp, err := altRT.RoundTrip(req); err != ErrSkipAltProtocol {
			return resp, err
		}

		if req.GetBody != nil {
			newReq := *req
			var ierr error
			newReq.Body, ierr = req.GetBody()
			if ierr != nil {
				return nil, ierr
			}
			req = &newReq
		}
	}

I do have a fix that returns Get "https://127.0.0.1:45927/put/0": http2: Transport received Server's graceful shutdown GOAWAY rather than the status code: 500, body: request declared a Content-Length of 5 but only wrote 0 bytes.

I don't think return an error is acceptable.

Change https://golang.org/cl/234358 mentions this issue: net/http: rewind the request body if alternate RoundTrip failed.

I think I had figured out what happening:

1. Alternate RoundTrip is configured "https" with golang.org/x/net/http2 by us.
2. http2 RoundTrip("https" Alternate RoundTrip) has sent request body but received a HTTP2 GOAWAY frame, and then failed re-try to get connection from connection pool. The error is wrapper as ErrSkipAltProtocol.
3. Then http RoundTrip take over request, and we do not re-set request body before that:

And I think #39086 (comment) would be a good solution.

/cc @bcmills

Is this a duplicate of #32441?

@bcmills yes it is.

Duplicate of #32441

Thanks @fraenkel @bcmills