I just tried on tip go version devel +867dcb5 Fri Jan 6 01:58:00 2017 +0000 darwin/amd64 and got the same result.

It's not a problem in the testing package. The net/http Server is seeing a bad request and closing the connection. You have a bunch of different goroutines, running in parallel, writing to the same network connection. As the socket buffer fills up, nothing guarantees that those writes are atomic. When I run an strace, I see that eventually a short write occurs. The Go code calls write again with the remaining bytes, but in the meantime another thread does a write to the same socket. The HTTP server sees a mangled request, returns a "400 Bad Request" error, and closes the connection. The next write fails with a broken pipe, and your code panics.

I don't think there is anything we can fix here, so closing.

I've modified the code to not share a connection but rather each running goroutine gets their own. I'm still seeing the issue and have dug around in dtruss looking for clues but I don't see any malformed requests expect when I copy the responses to stderr I do see a 400 response after many valid responses. I may be doing something very silly again but I can't seem to find it. Any help is appreciated.

Update bench:

package server

import (
	"io"
	"io/ioutil"
	"net"
	"net/http"
	"net/http/httptest"
	"testing"
)

func BenchmarkBug(b *testing.B) {
	hts := httptest.NewServer(http.HandlerFunc(http.NotFound))
	defer hts.Close()

	reqBytes := []byte("GET / HTTP/1.1\r\nHost: e.com\r\n\r\n")

	b.RunParallel(func(pb *testing.PB) {
		cn, err := net.Dial("tcp", hts.Listener.Addr().String())
		if err != nil {
			panic(err)
		}
		defer cn.Close()

		go io.Copy(ioutil.Discard, cn)

		for pb.Next() {
			_, err := cn.Write(reqBytes)
			if err != nil {
				panic(err)
			}
		}
	})
}

CC @bradfitz

This is a real bug. The net/http server is dropping an occasional byte.

If I add these lines to connReader.startBackgroundRead

	if cr.hasByte {
		panic("startBackgroundRead when already have byte")
	}

your program triggers the panic. Doing a background read when we already have a byte from an earlier background read is causing one of the bytes to be lost.

I don't understand exactly what triggers the condition. It has something to do with a full TCP buffer containing many GET requests and ending in the middle of a line.

I hope that Brad can take it from here.

I guess I now have weekend plans.

Your weekend is open again: https://golang.org/cl/34920.

CL https://golang.org/cl/34920 mentions this issue.

Killer. Thank you @ianlancetaylor.