Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: Excess found: excess = 2 url = / (zero-length body) in HEAD response #62015

Open
mmatczuk opened this issue Aug 14, 2023 · 1 comment
Open

net/http: Excess found: excess = 2 url = / (zero-length body) in HEAD response #62015

mmatczuk opened this issue Aug 14, 2023 · 1 comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@mmatczuk
Copy link

mmatczuk commented Aug 14, 2023 •
edited

What version of Go are you using (go version)?

go version go1.21.0 darwin/arm64

Does this issue reproduce with the latest release?

Yes

What did you do?

I'm writing a response to http HEAD request that contains chunked Transfer-Encoding.
The response as written contains additional "\r\n".

The program below demonstrates the behavior

package main

import (
	"bufio"
	"bytes"
	"io"
	"net"
	"net/http"
	"os"
	"os/exec"
	"testing"
)

func TestWriteHeadResponse(t *testing.T) {
	l, err := net.Listen("tcp", ":0")
	if err != nil {
		t.Fatal(err)
	}
	defer l.Close()

	go func() {
		conn, err := l.Accept()
		if err != nil {
			t.Fatal(err)
		}

		handleConn(t, conn)

		defer conn.Close()
	}()

	var buf bytes.Buffer

	cmd := exec.Command("curl", "-x", l.Addr().String(), "-v", "--head", "http://www.google.com")
	cmd.Stdout = os.Stdout
	cmd.Stderr = io.MultiWriter(&buf, os.Stderr)

	if err := cmd.Run(); err != nil {
		t.Fatal(err)
	}

	if bytes.Contains(buf.Bytes(), []byte("Excess found: excess = 2 url = / (zero-length body)")) {
		t.Fatal("excess found")
	}
}

func handleConn(t *testing.T, conn net.Conn) {
	brw := bufio.NewReadWriter(bufio.NewReader(conn), bufio.NewWriter(conn))
	defer brw.Flush()

	req, err := http.ReadRequest(brw.Reader)
	if err != nil {
		t.Fatal(err)
	}

	if req.Method != "HEAD" {
		t.Errorf("unexpected method: %s", req.Method)
	}

	res, err := http.DefaultTransport.RoundTrip(req)
	if err != nil {
		t.Fatal(err)
	}

	if err := res.Write(brw); err != nil {
		t.Fatal(err)
	}
}

Output:

=== RUN   TestWriteHeadResponse
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::]:53490...
* Connected to :: (::1) port 53490 (#0)
> HEAD http://www.google.com/ HTTP/1.1
> Host: www.google.com
> User-Agent: curl/8.1.2
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
< HTTP/1.1 200 OK
< Transfer-Encoding: chunked
< Cache-Control: private
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-2fpqee0gttwFKLELsSVWiA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Content-Type: text/html; charset=ISO-8859-1
< Date: Mon, 14 Aug 2023 14:28:16 GMT
< Expires: Mon, 14 Aug 2023 14:28:16 GMT
< Server: gws
< Set-Cookie: AEC=Ad49MVFERbRwKC9-DN6KNUJPfviWTjnnekKDlLJgfFbhCWrT6gCW7Ft63OA; expires=Sat, 10-Feb-2024 14:28:16 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 0
< 
* Excess found: excess = 2 url = / (zero-length body)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
HTTP/1.1 200 OK
Transfer-Encoding: chunked
Cache-Control: private
Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-2fpqee0gttwFKLELsSVWiA' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
Content-Type: text/html; charset=ISO-8859-1
Date: Mon, 14 Aug 2023 14:28:16 GMT
Expires: Mon, 14 Aug 2023 14:28:16 GMT
Server: gws
Set-Cookie: AEC=Ad49MVFERbRwKC9-DN6KNUJPfviWTjnnekKDlLJgfFbhCWrT6gCW7Ft63OA; expires=Sat, 10-Feb-2024 14:28:16 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

* Connection #0 to host :: left intact
    chunked_test.go:43: excess found
--- FAIL: TestWriteHeadResponse (0.33s)

FAIL

Note the Excess found: excess = 2 url = / (zero-length body)

If you add res.TransferEncoding = nil before writing the response, the test passes but the response comes with Connection: close.

Output:

=== RUN   TestWriteHeadResponse
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying [::]:53555...
* Connected to :: (::1) port 53555 (#0)
> HEAD http://www.google.com/ HTTP/1.1
> Host: www.google.com
> User-Agent: curl/8.1.2
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0< HTTP/1.1 200 OK
< Connection: close
< Cache-Control: private
< Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-JT0gk8j00rtWa2ER4JQozg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
< Content-Type: text/html; charset=ISO-8859-1
< Date: Mon, 14 Aug 2023 14:36:33 GMT
< Expires: Mon, 14 Aug 2023 14:36:33 GMT
< Server: gws
< Set-Cookie: AEC=Ad49MVFNMKsQRucydwAuOMpD55wbsCpuL8IRJwy8xM_oGNFHAspeS5ZMBpQ; expires=Sat, 10-Feb-2024 14:36:33 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
< X-Frame-Options: SAMEORIGIN
< X-Xss-Protection: 0
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Closing connection 0
HTTP/1.1 200 OK
Connection: close
Cache-Control: private
Content-Security-Policy-Report-Only: object-src 'none';base-uri 'self';script-src 'nonce-JT0gk8j00rtWa2ER4JQozg' 'strict-dynamic' 'report-sample' 'unsafe-eval' 'unsafe-inline' https: http:;report-uri https://csp.withgoogle.com/csp/gws/other-hp
Content-Type: text/html; charset=ISO-8859-1
Date: Mon, 14 Aug 2023 14:36:33 GMT
Expires: Mon, 14 Aug 2023 14:36:33 GMT
Server: gws
Set-Cookie: AEC=Ad49MVFNMKsQRucydwAuOMpD55wbsCpuL8IRJwy8xM_oGNFHAspeS5ZMBpQ; expires=Sat, 10-Feb-2024 14:36:33 GMT; path=/; domain=.google.com; Secure; HttpOnly; SameSite=lax
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 0

--- PASS: TestWriteHeadResponse (0.36s)
PASS

It makes it impossible to use http.Response::Write to implement

The HEAD method is identical to GET except that the server MUST NOT send content in the response. HEAD is used to obtain metadata about the selected representation without transferring its representation data, often for the sake of testing hypertext links or finding recent modifications.

https://www.rfc-editor.org/rfc/rfc9110.html#section-9.3.2

The code should write the readers and skip the body as specified.
@mmatczuk mmatczuk mentioned this issue Aug 14, 2023
Closed
mmatczuk added a commit to saucelabs/forwarder that referenced this issue Aug 14, 2023
@mmatczuk
martian: fix Excess found: excess = 2 url = / (zero-length body) in H…
32ff6be 
…EAD response

This works around golang/go#62015 by manually writing response to HEAD requests.

Fixes #357
@bcmills
Copy link
Contributor

bcmills commented Aug 14, 2023

(CC @neild)

@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 14, 2023
@bcmills bcmills added this to the Backlog milestone Aug 14, 2023
Choraden pushed a commit to saucelabs/forwarder that referenced this issue Aug 31, 2023
@mmatczuk @Choraden
martian: fix Excess found: excess = 2 url = / (zero-length body) in H…
93ad74f 
…EAD response

This works around golang/go#62015 by manually writing response to HEAD requests.

Fixes #357
Choraden pushed a commit to saucelabs/forwarder that referenced this issue Aug 31, 2023
@mmatczuk @Choraden
martian: fix Excess found: excess = 2 url = / (zero-length body) in H…
b7e80ea 
…EAD response

This works around golang/go#62015 by manually writing response to HEAD requests.

Fixes #357
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
2 participants
@mmatczuk @bcmills