Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

H2C Server causes Truncated Body On HTTP2 Requests with a Reverse Proxy, but only when writing a new response body. (Server Reset Stream) #42225

Closed
austincollinpena opened this issue Oct 27, 2020 · 2 comments

Comments

@austincollinpena
Copy link

austincollinpena commented Oct 27, 2020

What version of Go are you using (1.153)?

$ 1.153

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\Austin Pena\AppData\Local\go-build
set GOENV=C:\Users\Austin Pena\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\Austin Pena\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\AUSTIN~1\AppData\Local\Temp\go-build661771174=/tmp/go-build -gno-record-gcc-switches

What did you do?

I was getting timeouts on Go Playground otherwise I would have posted it there.

Here's the code for reproducibility. Please try to access https://easy-dp.ngrok.io to see the issue.

  1. Create a Reverse Proxy accessing Gzipped/ Br encoded Content
  2. Request a publicly available URL, I just grabbed Google Analytics
  3. Attempt to encode and decode the response via an http2 connection with a proxy.modifyresponse function
  4. Watch as content is dropped.

However, this only occurs under the following conditions:

  • Under SSL, like with https://easy-dp.ngrok.io
  • When running a proxy.ModifyResponse function
  • Decompressing and re-compressing the body (for example, just reading and rewriting the body to new bytes works)
package main

import (
	"bytes"
	"compress/gzip"
	"fmt"
	"golang.org/x/net/http2"
	"golang.org/x/net/http2/h2c"
	"io/ioutil"
	"net/http"
	"net/http/httputil"
	"strconv"
	"time"
)

func ForwardAnalytics(req *http.Request) {
	req.URL.Scheme = "https"
	req.URL.Host = "www.google-analytics.com"
	req.Host = "www.google-analytics.com"
	req.URL.Path = "/analytics.js"
	req.Header.Set("Accept-Encoding", "gzip")
}

func ModifyAnalytics(r *http.Response) error {
	bytesFromBody, err := ioutil.ReadAll(r.Body)
	defer r.Body.Close()
	if err != nil {
		return nil
	}
	if r.Header.Get("Content-Encoding") == "gzip" {
		gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
		if err != nil {
			return nil
		}
		defer gzipReader.Close()
		readableBytes, err := ioutil.ReadAll(gzipReader)
		var b bytes.Buffer
		gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
		if err != nil {
			return nil
		}
		defer gzipWriter.Close()
		writtenLen, err := gzipWriter.Write(readableBytes)
		fmt.Println("Wrote ", writtenLen)
		if err != nil {
			return nil
		}
		r.ContentLength = int64(len(readableBytes))
		r.Header.Set("Content-Length", strconv.FormatInt(int64(len(readableBytes)), 10))
		r.Body = ioutil.NopCloser(&b)

		return nil
	} else {
		return nil
	}
}


func handleProxy(w http.ResponseWriter, req *http.Request) {
	proxy := httputil.ReverseProxy{
		Director: ForwardAnalytics
	}
	proxy.ModifyResponse = ModifyAnalytics
	proxy.ServeHTTP(w, req)

}

func main() {
	h2s := &http2.Server{
		IdleTimeout: 20 * time.Second,
	}
	mux := http.NewServeMux()
	mux.HandleFunc( "/", handleProxy)
	s := &http.Server{
		ReadHeaderTimeout: 20 * time.Second,
		ReadTimeout:       10 * time.Second,
		WriteTimeout:      30 * time.Second,
		Addr:              "localhost:8456",
		Handler:           h2c.NewHandler(mux, h2s),
	}
	s.ListenAndServe()
}

What did you expect to see?

I expect to see the ability to open the bytes, modify them, and update the response body on an H2C connection

What did you see instead?

Two things of note happen:

  1. Chrome gives a nice little error that expands upon what's going on
{"params":{"description":"Server reset stream.","net_error":"ERR_HTTP2_PROTOCOL_ERROR","stream_id":5},"phase":0,"source":{"id":1493828,"start_time":"732370299","type":1},"time":"732375561","type":224},
  1. Under the normal http connection, there's no problem, but under the https connection the script may or may not print out to a certain length. Sometimes it doesn't print at all, sometimes it prints about 30%.

This is a cross browser issue.

@seankhliao
Copy link
Member

The Content-Length header indicates the size of the entity body in the message, in bytes. The size includes any content encodings (the Content-Length of a gzip-compressed text file will be the compressed size, not the original size).

@austincollinpena
Copy link
Author

@seankhliao Thanks for taking the time to point that out. I had originally done that, but failed to close the gzip writer correctly. I was having this issue until I rewrote my code to the following:

	if r.Header.Get("Content-Encoding") == "gzip" {
		gzipReader, err := gzip.NewReader(bytes.NewBuffer(bytesFromBody))
		if err != nil {
			return nil
		}
		defer gzipReader.Close()
		readableBytes, err := ioutil.ReadAll(gzipReader)
		var b bytes.Buffer
		gzipWriter, err := gzip.NewWriterLevel(&b, gzip.DefaultCompression)
		if err != nil {
			return nil
		}
		writtenLen, err := gzipWriter.Write(readableBytes)
		gzipWriter.Close() // This was the culprit. It needed to be closed here
		fmt.Println("Wrote ", writtenLen)
		if err != nil {
			return nil
		}
		r.ContentLength = int64(b.Len())
		r.Header.Set("Content-Length", strconv.FormatInt(int64(b.Len()), 10))
		r.Body = ioutil.NopCloser(&b)
		return nil
	}

It's interesting to me that it only happened over an H2C connection though. Perhaps it was just due to latency? Hard to say.

@golang golang locked and limited conversation to collaborators Oct 27, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants