Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/net/http2: make net/http/httputil/ DumpRequestOut working on http2 ?? #18464

Closed
c0b opened this issue Dec 29, 2016 · 8 comments
Closed

x/net/http2: make net/http/httputil/ DumpRequestOut working on http2 ?? #18464

c0b opened this issue Dec 29, 2016 · 8 comments

Comments

@c0b
Copy link

c0b commented Dec 29, 2016

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.7.4

What operating system and processor architecture are you using (go env)?

Linux amd64

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

A simple http2 client program, make get request to https://http2bin.org/get

use DumpResponse / DumpRequestOut (from net/http/httputil/) to print the Request & Response,

func main() {
	const url = "https://http2bin.org/get"
	resp, _ := http.Get(url)

	dump, _ := httputil.DumpRequestOut(resp.Request, true)
	os.Stdout.Write(dump)

	dump, _ = httputil.DumpResponse(resp, true)

	os.Stdout.Write(dump)
}

What did you expect to see?

http2 headers for request & response should be different with http1, should be all lower cases, in Chrome DevTool like other command line tools it's commonly shown as:

:authority: http2bin.org
:method: GET
:path: /get
:scheme: https
accept-encoding: gzip
user-agent: ...

[request body]

:status: 200
server: h2o/2.0.4
date: ...
content-type: ...

[response body]

What did you see instead?

GET /get HTTP/1.1
Host: http2bin.org
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

HTTP/2.0 200 OK
Content-Length: 268
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Thu, 29 Dec 2016 09:24:32 GMT
Server: h2o/2.0.4
X-Clacks-Overhead: GNU Terry Pratchett

{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Connection": "keep-alive", 
    "Host": "http2bin.org", 
    "User-Agent": "Go-http-client/2.0", 
    "Via": "2 http2bin.org"
  }, 
  "origin": "XXXXXXXXX", 
  "url": "https://http2bin.org/get"
}

========

BTW, set GODEBUG=http2debug=2 get following debugging logs:

$ time GODEBUG=http2debug=2 go run http2ie.go
2016/12/29 01:28:05 http2: Transport failed to get client conn for http2bin.org:443: http2: no cached connection was available
2016/12/29 01:28:06 http2: Transport creating client conn 0xc420462680 to 172.99.78.199:443
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: wrote SETTINGS len=18, settings: ENABLE_PUSH=0, INITIAL_WINDOW_SIZE=4194304, MAX_HEADER_LIST_SIZE=10485760
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: wrote WINDOW_UPDATE len=4 (conn) incr=1073741824
2016/12/29 01:28:06 http2: Transport encoding header ":authority" = "http2bin.org"
2016/12/29 01:28:06 http2: Transport encoding header ":method" = "GET"
2016/12/29 01:28:06 http2: Transport encoding header ":path" = "/get"
2016/12/29 01:28:06 http2: Transport encoding header ":scheme" = "https"
2016/12/29 01:28:06 http2: Transport encoding header "accept-encoding" = "gzip"
2016/12/29 01:28:06 http2: Transport encoding header "user-agent" = "Go-http-client/2.0"
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: wrote HEADERS flags=END_STREAM|END_HEADERS stream=1 len=38
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: read SETTINGS len=12, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=16777216
2016/12/29 01:28:06 http2: Transport received SETTINGS len=12, settings: MAX_CONCURRENT_STREAMS=100, INITIAL_WINDOW_SIZE=16777216
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: wrote SETTINGS flags=ACK len=0
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: read SETTINGS flags=ACK len=0
2016/12/29 01:28:06 http2: Transport received SETTINGS flags=ACK len=0
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: read HEADERS flags=END_HEADERS stream=1 len=115
2016/12/29 01:28:06 http2: decoded hpack field header field ":status" = "200"
2016/12/29 01:28:06 http2: decoded hpack field header field "server" = "h2o/2.0.4"
2016/12/29 01:28:06 http2: decoded hpack field header field "date" = "Thu, 29 Dec 2016 09:28:06 GMT"
2016/12/29 01:28:06 http2: decoded hpack field header field "content-type" = "application/json"
2016/12/29 01:28:06 http2: decoded hpack field header field "access-control-allow-origin" = "*"
2016/12/29 01:28:06 http2: decoded hpack field header field "access-control-allow-credentials" = "true"
2016/12/29 01:28:06 http2: decoded hpack field header field "x-clacks-overhead" = "GNU Terry Pratchett"
2016/12/29 01:28:06 http2: decoded hpack field header field "content-length" = "268"
2016/12/29 01:28:06 http2: Transport received HEADERS flags=END_HEADERS stream=1 len=115
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: read DATA stream=1 len=267 data="{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept-Encoding\": \"gzip\", \n    \"Connection\": \"keep-alive\", \n    \"Host\": \"http2bin.org\", \n    \"User-Agent\": \"Go-http-client/2.0\", \n    \"Via\": \"2 http2bin.org\"\n  }, \n  \"origin\": \"XXXXXXXXX\", \n  \"url\": \"https://http2bin" (11 bytes omitted)
2016/12/29 01:28:06 http2: Transport received DATA stream=1 len=267 data="{\n  \"args\": {}, \n  \"headers\": {\n    \"Accept-Encoding\": \"gzip\", \n    \"Connection\": \"keep-alive\", \n    \"Host\": \"http2bin.org\", \n    \"User-Agent\": \"Go-http-client/2.0\", \n    \"Via\": \"2 http2bin.org\"\n  }, \n  \"origin\": \"XXXXXXXXX\", \n  \"url\": \"https://http2bin" (11 bytes omitted)
2016/12/29 01:28:06 http2: Framer 0xc42012e0c0: read DATA flags=END_STREAM stream=1 len=1 data="\n"
2016/12/29 01:28:06 http2: Transport received DATA flags=END_STREAM stream=1 len=1 data="\n"
GET /get HTTP/1.1
Host: http2bin.org
User-Agent: Go-http-client/1.1
Accept-Encoding: gzip

HTTP/2.0 200 OK
Content-Length: 268
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Content-Type: application/json
Date: Thu, 29 Dec 2016 09:28:06 GMT
Server: h2o/2.0.4
X-Clacks-Overhead: GNU Terry Pratchett

{
  "args": {}, 
  "headers": {
    "Accept-Encoding": "gzip", 
    "Connection": "keep-alive", 
    "Host": "http2bin.org", 
    "User-Agent": "Go-http-client/2.0", 
    "Via": "2 http2bin.org"
  }, 
  "origin": "XXXXXXXXX", 
  "url": "https://http2bin.org/get"
}

real	0m1.416s
user	0m0.488s
sys	0m0.160s
@c0b
Copy link
Author

c0b commented Dec 29, 2016

from Chrome DevTool:

image

@c0b
Copy link
Author

c0b commented Dec 29, 2016

the actual need behind this is I want some application level debugging for http2, from https://golang.org/pkg/net/http/httptrace/ it's good to have a WroteHeaders hook point, but that only answered a question of when all headers are written, it's when my callback is called, but its signature is WroteHeaders func() it didn't answer a question of what headers are written? I want a copy of what's going on to the wire;

from the (cc *http2ClientConn) RoundTrip function we can see there is a local var hdrs have a hold of all written headers in []byte format, I guess it's hpack compressed already,
can I request WroteHeaders signature changed to be called with all uncompressed headers?

https://golang.org/src/net/http/h2_bundle.go #L5496

  5496	hdrs, err := cc.encodeHeaders(req, requestedGzip, trailers, contentLen)

https://github.com/golang/net/blob/master/http2/transport.go#L754-L772

	// we send: HEADERS{1}, CONTINUATION{0,} + DATA{0,} (DATA is
	// sent by writeRequestBody below, along with any Trailers,
	// again in form HEADERS{1}, CONTINUATION{0,})
	hdrs, err := cc.encodeHeaders(req, requestedGzip, trailers, contentLen)
	if err != nil {
		cc.mu.Unlock()
		return nil, err
	}

	cs := cc.newStream()
	cs.req = req
	cs.trace = requestTrace(req)
	cs.requestedGzip = requestedGzip
	bodyWriter := cc.t.getBodyWriterState(cs, body)
	cs.on100 = bodyWriter.on100

	cc.wmu.Lock()
	endStream := !hasBody && !hasTrailers
	werr := cc.writeHeaders(cs.ID, endStream, hdrs)

@bradfitz
Copy link
Contributor

For questions about Go, see https://golang.org/wiki/Questions.

As the docs say:

HTTP/2 requests are dumped in HTTP/1.x form, not in their original binary representations.

Dumping in binary would be unreadable and useless.

If you want to see how the sausage is made, see https://godoc.org/golang.org/x/net/http2.

@c0b
Copy link
Author

c0b commented Dec 30, 2016

why is this closed so early? @bradfitz

I didn't ask to dump the binary, but a better text format more close to what's seen in Chrome DevTool:

like

:authority: http2bin.org
:method: GET
:path: /get
:scheme: https
accept-encoding: gzip
user-agent: ...

[request body]

:status: 200
server: h2o/2.0.4
date: ...
content-type: ...

[response body]

@c0b
Copy link
Author

c0b commented Dec 30, 2016

this is not only a question, but something deserve to be improved in code

see https://godoc.org/golang.org/x/net/http2.

of course I have read that; but the problem I pointed out is the current http2 code didn't give application level an inspect interface to know what headers are written into the wire; this is a feature request

@bradfitz
Copy link
Contributor

The Chrome text format is almost as arbitrary as any http2-binary-to-text format, including Go's format, which is at least consistent with Go's http1 format.

If you want a certain format, you're free to implement one. I don't think more formats are a good idea for the standard library. It's too little value for too much overlap.

I'd prefer this be implemented outside the Go libraries as an opt-in mechanism people can use when they want to match Chrome.

We already provide GODEBUG=http2debug=2 when you want to see exactly what's happening.

@c0b
Copy link
Author

c0b commented Dec 30, 2016

the GODEBUG=http2debug=2 is not enough because I have read this part of code it's just setting internal variables and print to log with extra information, in fixed format like I have shown above; but in a user friendly application design, I need to show the detail http2 interaction in different format, could be in a GUI, or saved in key=value pairs in Database for lookup later, from Application design level, need better and flexible interfaces

https://github.com/golang/net/blob/master/http2/http2.go#L42-L52

I have found the httptrace package could be a good starting point, but it's also very limited, I can register a WroteHeaders callback to be notified when the headers are actually wrote, at that moment with this interface I can only do is to record a timestamp; but what are the headers written?

WroteRequest is called with WroteRequestInfo looks like a little better,

https://golang.org/pkg/net/http/httptrace/

        // WroteHeaders is called after the Transport has written
        // the request headers.
        WroteHeaders func()

        // WroteRequest is called with the result of writing the
        // request and any body.
        WroteRequest func(WroteRequestInfo)

@c0b
Copy link
Author

c0b commented Dec 30, 2016

if httputil is not agreed to change to show http2 traffic in different way, that is fine, I would like to do differentiation in Application level; then a different issue can be opened for this as a feature request? because currently many fields on http2ClientConn struct are lower cased (private); this library users won't be able to access, those can be changed to public (not good), or this library could design with proper API to expose such information when users want

Let me point you some different API design in different pro language / libraries: @bradfitz

  1. nghttp2 in C; notice in their tutorial client program, users can register a on_header_callback function, this function's signature shows it will get name, value pairs in the order headers are received
    http://nghttp2.org/documentation/tutorial-client.html
    http://nghttp2.org/documentation/nghttp2_session_callbacks_set_on_header_callback.html

  2. nodejs' http2 in core; users are freely to listen events like 'begin-headers', 'header' (for each name, value pair on the order they are received), 'headers-complete', 'stream-close', ... many of them; because this code is utilizing nghttp2 as underlayer and is just a thin wrapper
    https://github.com/nodejs/http2/blob/master/doc/http2-implementation-notes.md#event-header

Current Golang http2 code doesn't have any of these features; I have read the doc and most of the Go code under net/http2
https://godoc.org/golang.org/x/net/http2 https://github.com/golang/net/tree/master/http2

@golang golang locked and limited conversation to collaborators Dec 30, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants