Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net/http/httputil: ReverseProxy add a API to get the length of proxy had sent data #57777

Open
lwwgo opened this issue Jan 13, 2023 · 4 comments
Labels
Milestone

Comments

@lwwgo
Copy link

lwwgo commented Jan 13, 2023

demo code:

revProxy := httputil.NewSingleHostReverseProxy(url)
  modifyFunc := func(res *http.Response) error {
		body, err := io.ReadAll(res.Body)
		res.Body = io.NopCloser(bytes.NewReader(body))
		if err != nil {
			return err
		}
		count := len(body)
		return nil
	}
	revProxy.ModifyResponse = modifyFunc

goal:get the length of proxy had sent data, it is successful, but it is block when response is live streaming, because io.ReadAll() can not read EOF or error from streaming。The stream has always existed and has no end。

Design:

type ReverseProxy struct {
    SendBodySize atomic.Int64
}
// copyBuffer returns any write errors or non-EOF read errors, and the amount
// of bytes written.
func (p *ReverseProxy) copyBuffer(dst io.Writer, src io.Reader, buf []byte) (int64, error) {
	if len(buf) == 0 {
		buf = make([]byte, 32*1024)
	}
	var written int64
	for {
		nr, rerr := src.Read(buf)
		if rerr != nil && rerr != io.EOF && rerr != context.Canceled {
			p.logf("httputil: ReverseProxy read error during body copy: %v", rerr)
		}
		if nr > 0 {
			nw, werr := dst.Write(buf[:nr])
			if nw > 0 {
				written += int64(nw)
                                 p.SendBodySize.Add(int64(nw))  // add code
			}
			if werr != nil {
				return written, werr
			}
			if nr != nw {
				return written, io.ErrShortWrite
			}
		}
		if rerr != nil {
			if rerr == io.EOF {
				rerr = nil
			}
			return written, rerr
		}
	}
}

func (p *ReverseProxy) ResetSendSize() {
    p.SendBodySize.Store(0)
}
@lwwgo lwwgo added the Proposal label Jan 13, 2023
@gopherbot gopherbot added this to the Proposal milestone Jan 13, 2023
@lwwgo lwwgo changed the title proposal: http/httputil: ReverseProxy add a API to get the length of proxy had send data proposal: http/httputil: ReverseProxy add a API to get the length of proxy had sent data Jan 13, 2023
@seankhliao seankhliao changed the title proposal: http/httputil: ReverseProxy add a API to get the length of proxy had sent data proposal: net/http/httputil: ReverseProxy add a API to get the length of proxy had sent data Jan 13, 2023
@seankhliao
Copy link
Member

This functionality looks orthogonal to the purpose of ReverseProxy (you could just as easily wrap it in middleware that will do the tracking for you, eg https://pkg.go.dev/github.com/felixge/httpsnoop).
Additionally, a per-proxy count doesn't appear to be very useful, most people seem to want a per handler count.

@ianlancetaylor
Copy link
Contributor

CC @neild @bradfitz

@lwwgo
Copy link
Author

lwwgo commented Jan 14, 2023

This functionality looks orthogonal to the purpose of ReverseProxy (you could just as easily wrap it in middleware that will do the tracking for you, eg https://pkg.go.dev/github.com/felixge/httpsnoop). Additionally, a per-proxy count doesn't appear to be very useful, most people seem to want a per handler count.

thanks for your response. felixge/httpsnoop package:

// myH is your app's http handler, perhaps a http.ServeMux or similar.
var myH http.Handler
// wrappedH wraps myH in order to log every request.
wrappedH := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
	m := httpsnoop.CaptureMetrics(myH, w, r)   // This line is block, when header contains "Transfer-Encoding: chunked"
	log.Printf(
		"%s %s (code=%d dt=%s written=%d)",
		r.Method,
		r.URL,
		m.Code,
		m.Duration,
		m.Written,
	)
})
http.ListenAndServe(":8080", wrappedH)

It does not work, when ReverseProxy is used for proxying live streaming by http-flv format and backend http server response data by chunk mode, in other words, Transfer-Encoding: chunked.
Unfortunately, CaptureMetrics() does not return any statistics until the live stream is closed. Obviously, felixge/httpsnoop package is ineffective for live stream proxy. I need a golang packet with real-time statistics of forwarding traffic.
Do you have other solutions? @seankhliao
Thanks.

@neild
Copy link
Contributor

neild commented Jan 14, 2023

As @seankhliao says, this seems orthogonal to ReverseProxy. You could as easily want to track bytes sent by some other handler.

You should be able to build this with a middleware handler that wraps the http.ResponseWriter. Something like:

type CountingResponseWriter struct {
  http.ResponseWriter
  Count int
}

func (c *CountingResponseWriter) Write(p []byte) (n int, err error) {
  n, err = c.ResponseWriter.Write(p)
  c.Count += n // apply locking as needed
  return n, err
}

func (c *CountingResponseWriter) Unwrap() http.ResponseWriter { return c.ResponseWriter }

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

5 participants