Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: TestTransportPersistConnLeak test failure/flake on windows 2003 #14375

Closed
litg opened this issue Feb 18, 2016 · 7 comments
Closed

net/http: TestTransportPersistConnLeak test failure/flake on windows 2003 #14375

litg opened this issue Feb 18, 2016 · 7 comments
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Milestone

Comments

@litg
Copy link

litg commented Feb 18, 2016

ok net 23.938s
--- FAIL: TestTransportPersistConnLeak (0.17s)
transport_test.go:1032: goroutine growth: 24 -> 121 -> 36 (delta: 12)
transport_test.go:1033: too many new goroutines
FAIL
FAIL net/http 19.625s
ok net/http/cgi 2.000s

@ianlancetaylor ianlancetaylor changed the title failure to build go 1.6 on windows 2003 net/http: failure to build go 1.6 on windows 2003 Feb 18, 2016
@ianlancetaylor ianlancetaylor added this to the Go1.6.1 milestone Feb 18, 2016
@ianlancetaylor
Copy link
Contributor

CC @bradfitz @alexbrainman

Note that you can still use your Go installation, despite this test failure.

@bradfitz
Copy link
Contributor

Hm. I haven't seen that one, or at least not in ages. I wonder what they are.

@bradfitz bradfitz self-assigned this Feb 18, 2016
@bradfitz bradfitz changed the title net/http: failure to build go 1.6 on windows 2003 net/http: TestTransportPersistConnLeak test failure/flake on windows 2003 Feb 18, 2016
@litg
Copy link
Author

litg commented Feb 18, 2016

transport_test.go

line 1022:
-- time.Sleep(100 * time.Millisecond)
++ time.Sleep(200 * time.Millisecond)

ALL TESTS PASSED

@alexbrainman
Copy link
Member

I can reproduce this:

C:\dev\go\src\net\http>go version
go version devel +a4b8339 Mon Feb 22 21:15:50 2016 +0000 windows/386

C:\dev\go\src\net\http>git diff
diff --git a/src/net/http/transport_test.go b/src/net/http/transport_test.go
index d9da078..12b1862 100644
--- a/src/net/http/transport_test.go
+++ b/src/net/http/transport_test.go
@@ -968,6 +968,12 @@ func TestTransportGzipShort(t *testing.T) {
    }
 }

+func captureStack() string {
+   buf := make([]byte, 1<<16)
+   n := runtime.Stack(buf, true)
+   return string(buf[:n])
+}
+
 // tests that persistent goroutine connections shut down when no longer desired.
 func TestTransportPersistConnLeak(t *testing.T) {
    setParallel(t)
@@ -986,6 +992,7 @@ func TestTransportPersistConnLeak(t *testing.T) {
    c := &Client{Transport: tr}

    n0 := runtime.NumGoroutine()
+   stack0 := captureStack()

    const numReq = 25
    didReqCh := make(chan bool)
@@ -1023,9 +1030,14 @@ func TestTransportPersistConnLeak(t *testing.T) {
    runtime.GC()
    runtime.GC() // even more.
    nfinal := runtime.NumGoroutine()
+   stackfinal := captureStack()

    growth := nfinal - n0

+   fmt.Printf("%v\n", stack0)
+   fmt.Printf("---------\n")
+   fmt.Printf("%v\n", stackfinal)
+
    // We expect 0 or 1 extra goroutine, empirically.  Allow up to 5.
    // Previously we were leaking one per numReq.
    if int(growth) > 5 {

C:\dev\go\src\net\http>go test -short
goroutine 5399 [running]:
net/http_test.captureStack(0x0, 0x0)
    c:/dev/go/src/net/http/transport_test.go:973 +0x7c
net/http_test.TestTransportPersistConnLeak(0x11112360)
    c:/dev/go/src/net/http/transport_test.go:995 +0x234
testing.tRunner(0x11112360, 0xa4ac20)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 1 [chan receive]:
testing.RunTests(0x8fc0bc, 0xa49e40, 0x155, 0x155, 0x40e001)
    c:/dev/go/src/testing/testing.go:603 +0x461
testing.(*M).Run(0x11109f7c, 0x360018)
    c:/dev/go/src/testing/testing.go:515 +0x79
net/http_test.TestMain(0x11109f7c)
    c:/dev/go/src/net/http/main_test.go:22 +0x21
main.main()
    net/http/_test/_testmain.go:778 +0x100

goroutine 58 [chan receive, locked to thread]:
net.(*ioSrv).ProcessRemoteIO(0x10c1ab30)
    c:/dev/go/src/net/fd_windows.go:139 +0xcc
created by net.startServer
    c:/dev/go/src/net/fd_windows.go:239 +0xc9

goroutine 59 [chan receive, locked to thread]:
net.(*ioSrv).ProcessRemoteIO(0x10c1ab38)
    c:/dev/go/src/net/fd_windows.go:139 +0xcc
created by net.startServer
    c:/dev/go/src/net/fd_windows.go:241 +0x11e

goroutine 5401 [chan receive]:
testing.(*T).Parallel(0x111123c0)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x111123c0)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportPersistConnLeakShortBody(0x111123c0)
    c:/dev/go/src/net/http/transport_test.go:1052 +0x4e
testing.tRunner(0x111123c0, 0xa4ac2c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5529 [chan receive]:
testing.(*T).Parallel(0x11112ea0)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112ea0)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestCancelRequestWithChannel(0x11112ea0)
    c:/dev/go/src/net/http/transport_test.go:1575 +0x68
testing.tRunner(0x11112ea0, 0xa4acb0)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5306 [sleep]:
time.Sleep(0x17d7840, 0x0)
    c:/dev/go/src/runtime/time.go:59 +0xe6
net/http_test.TestTransportServerClosingUnexpectedly(0x11112060)
    c:/dev/go/src/net/http/transport_test.go:483 +0x205
testing.tRunner(0x11112060, 0xa4aba8)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5744 [IO wait]:
net.runtime_pollWait(0x31293658, 0x72, 0x112cbe14)
    c:/dev/go/src/runtime/netpoll.go:160 +0x55
net.(*pollDesc).Wait(0x10d66c30, 0x72, 0x0, 0x0)
    c:/dev/go/src/net/fd_poll_runtime.go:73 +0x35
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10d66b98, 0x847468, 0x8, 0x112cbdd0, 0x10c8e380, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:183 +0x122
net.(*netFD).acceptOne(0x10d66b60, 0x10c8e2a0, 0x2, 0x2, 0x10d66b98, 0x1114ff0c, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:583 +0x1d4
net.(*netFD).accept(0x10d66b60, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:613 +0x11f
net.(*TCPListener).AcceptTCP(0x10d89430, 0x1114ff44, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:254 +0x47
net.(*TCPListener).Accept(0x10d89430, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:264 +0x39
net/http.(*Server).Serve(0x112e2e10, 0x31292680, 0x10d89430, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:2117 +0x100
net/http/httptest.(*Server).goServe.func1(0x112e4d80)
    c:/dev/go/src/net/http/httptest/server.go:237 +0x66
created by net/http/httptest.(*Server).goServe
    c:/dev/go/src/net/http/httptest/server.go:238 +0x50

goroutine 5307 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112060)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5527 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112d80)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5532 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112f00)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5764 [runnable, locked to thread]:
syscall.Syscall(0x7c80f1c5, 0x3, 0x10c89c60, 0x111d15f0, 0x64, 0x64, 0x0, 0x0)
    c:/dev/go/src/runtime/syscall_windows.go:128 +0x4a
syscall.GetEnvironmentVariable(0x10c89c60, 0x111d15f0, 0x64, 0x64, 0x0, 0x0)
    c:/dev/go/src/syscall/zsyscall_windows.go:802 +0x67
syscall.Getenv(0x82b770, 0x7, 0x0, 0x0, 0x1)
    c:/dev/go/src/syscall/env_windows.go:22 +0xc5
os.Getenv(0x82b770, 0x7, 0x0, 0x0)
    c:/dev/go/src/os/env.go:80 +0x33
net/http.(*Server).onceSetNextProtoDefaults(0x112e2eb0)
    c:/dev/go/src/net/http/server.go:2290 +0x2e
net/http.(*Server).(net/http.onceSetNextProtoDefaults)-fm()
    c:/dev/go/src/net/http/server.go:2282 +0x20
sync.(*Once).Do(0x112e2ee8, 0x10e3bf30)
    c:/dev/go/src/sync/once.go:44 +0xcf
net/http.(*Server).setupHTTP2(0x112e2eb0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:2282 +0x47
net/http.(*Server).Serve(0x112e2eb0, 0x31292680, 0x10d894d0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:2113 +0xbd
net/http/httptest.(*Server).goServe.func1(0x112e5300)
    c:/dev/go/src/net/http/httptest/server.go:237 +0x66
created by net/http/httptest.(*Server).goServe
    c:/dev/go/src/net/http/httptest/server.go:238 +0x50

goroutine 5530 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112ea0)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5400 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112360)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5524 [chan receive]:
testing.(*T).Parallel(0x11112d20)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112d20)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportResponseHeaderTimeout(0x11112d20)
    c:/dev/go/src/net/http/transport_test.go:1393 +0xc8
testing.tRunner(0x11112d20, 0xa4ac8c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5402 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x111123c0)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5679 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x10c48d80)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5531 [chan receive]:
testing.(*T).Parallel(0x11112f00)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112f00)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestCancelRequestWithChannelBeforeDo(0x11112f00)
    c:/dev/go/src/net/http/transport_test.go:1633 +0x39
testing.tRunner(0x11112f00, 0xa4acbc)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5525 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112d20)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5526 [chan receive]:
testing.(*T).Parallel(0x11112d80)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112d80)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportCancelRequest(0x11112d80)
    c:/dev/go/src/net/http/transport_test.go:1465 +0x68
testing.tRunner(0x11112d80, 0xa4ac98)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5678 [chan receive]:
testing.(*T).Parallel(0x10c48d80)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x10c48d80)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportClosesBodyOnError(0x10c48d80)
    c:/dev/go/src/net/http/transport_test.go:2509 +0x5e
testing.tRunner(0x10c48d80, 0xa4ad7c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

---------
goroutine 5399 [running]:
net/http_test.captureStack(0x0, 0x0)
    c:/dev/go/src/net/http/transport_test.go:973 +0x7c
net/http_test.TestTransportPersistConnLeak(0x11112360)
    c:/dev/go/src/net/http/transport_test.go:1033 +0x3a9
testing.tRunner(0x11112360, 0xa4ac20)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 1 [chan receive]:
testing.RunTests(0x8fc0bc, 0xa49e40, 0x155, 0x155, 0x40e001)
    c:/dev/go/src/testing/testing.go:603 +0x461
testing.(*M).Run(0x11128f84, 0x360018)
    c:/dev/go/src/testing/testing.go:515 +0x79
net/http_test.TestMain(0x11128f84)
    c:/dev/go/src/net/http/main_test.go:22 +0x21
main.main()
    net/http/_test/_testmain.go:778 +0x100

goroutine 4 [runnable, locked to thread]:
syscall.Syscall(0x71ab3e2b, 0x1, 0x410, 0x0, 0x0, 0xa4b7c0, 0x31292c80, 0x6624ae)
    c:/dev/go/src/runtime/syscall_windows.go:128 +0x4a
syscall.Closesocket(0x410, 0x0, 0x0)
    c:/dev/go/src/syscall/zsyscall_windows.go:1488 +0x5f
net.(*netFD).destroy(0x10df1ea0)
    c:/dev/go/src/net/fd_windows.go:392 +0x4e
net.(*netFD).decref(0x10df1ea0)
    c:/dev/go/src/net/fd_windows.go:411 +0x3d
net.(*netFD).Close(0x10df1ea0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:453 +0x7c

goroutine 58 [syscall, locked to thread]:
syscall.Syscall(0x7c82f122, 0x1, 0x32c, 0x0, 0x0, 0x10da78c0, 0x10e41fbc, 0x663401)
    c:/dev/go/src/runtime/syscall_windows.go:128 +0x4a
syscall.CancelIo(0x32c, 0x0, 0x0)
    c:/dev/go/src/syscall/zsyscall_windows.go:550 +0x5f
net.(*ioSrv).ProcessRemoteIO(0x10c1ab30)
    c:/dev/go/src/net/fd_windows.go:143 +0x13f
created by net.startServer
    c:/dev/go/src/net/fd_windows.go:239 +0xc9

goroutine 59 [chan receive, locked to thread]:
net.(*ioSrv).ProcessRemoteIO(0x10c1ab38)
    c:/dev/go/src/net/fd_windows.go:139 +0xcc
created by net.startServer
    c:/dev/go/src/net/fd_windows.go:241 +0x11e

goroutine 5401 [sleep]:
time.Sleep(0x17d78400, 0x0)
    c:/dev/go/src/runtime/time.go:59 +0xe6
net/http_test.TestTransportPersistConnLeakShortBody(0x111123c0)
    c:/dev/go/src/net/http/transport_test.go:1076 +0x4fc
testing.tRunner(0x111123c0, 0xa4ac2c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5921 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8fb58, 0x82c470, 0x7, 0x8fc0e4, 0x100, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10c8fb20, 0x111ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10dd89c8, 0x111ba000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10dd89c8, 0x10d06ce4, 0x111ba000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10c88860, 0x111ba000, 0x1000, 0x1000, 0x2d0250, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x11241ec0)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x11241ec0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06cb0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5529 [chan receive]:
testing.(*T).Parallel(0x11112ea0)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112ea0)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestCancelRequestWithChannel(0x11112ea0)
    c:/dev/go/src/net/http/transport_test.go:1575 +0x68
testing.tRunner(0x11112ea0, 0xa4acb0)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5957 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e21538, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e21500, 0x10dae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa2c0, 0x10dae000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10cfa2c0, 0x10e0dae4, 0x10dae000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0be0, 0x10dae000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17e90)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17e90, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0dab0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5959 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10db0038, 0x82c470, 0x7, 0x8fc0e4, 0x111cd360, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:165 +0x58d
net.(*netFD).Read(0x10db0000, 0x10dea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa328, 0x10dea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.(*connReader).Read(0x111cd520, 0x10dea000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:526 +0x189
bufio.(*Reader).fill(0x10dfb8f0)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).ReadSlice(0x10dfb8f0, 0xa, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:328 +0x1b4
bufio.(*Reader).ReadLine(0x10dfb8f0, 0x0, 0x0, 0x0, 0x81bd00, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:357 +0x4b
net/textproto.(*Reader).readLineSlice(0x111cd540, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/textproto/reader.go:55 +0x59
net/textproto.(*Reader).ReadLine(0x111cd540, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/textproto/reader.go:36 +0x39
net/http.readRequest(0x10dfb8f0, 0x0, 0x10e0db20, 0x0, 0x0)
    c:/dev/go/src/net/http/request.go:721 +0x9a
net/http.(*conn).readRequest(0x111c5f40, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:705 +0x3b7
net/http.(*conn).serve(0x111c5f40)
    c:/dev/go/src/net/http/server.go:1425 +0x985
created by net/http.(*Server).Serve
    c:/dev/go/src/net/http/server.go:2137 +0x394

goroutine 5927 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8fd18, 0x82c470, 0x7, 0x8fc0e4, 0x100, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10c8fce0, 0x111be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10dd8ae8, 0x111be000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10dd8ae8, 0x10d06ea4, 0x111be000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10c88950, 0x111be000, 0x1000, 0x1000, 0x2d0250, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x11241f80)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x11241f80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06e70)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5948 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e21298, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e21260, 0x10cfe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa110, 0x10cfe000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10cfa110, 0x10e0d844, 0x10cfe000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0a70, 0x10cfe000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17d70)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17d70, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d810)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5942 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e210d8, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e210a0, 0x10cf8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10d89ff0, 0x10cf8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10d89ff0, 0x10e0d684, 0x10cf8000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0980, 0x10cf8000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17c80)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17c80, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d650)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5893 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e209d8, 0x847468, 0x8, 0x10ca0c20, 0x10e216c0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:165 +0x58d
net.(*netFD).acceptOne(0x10e209a0, 0x10e215e0, 0x2, 0x2, 0x10e209d8, 0x112d6f0c, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:583 +0x1d4
net.(*netFD).accept(0x10e209a0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:613 +0x11f
net.(*TCPListener).AcceptTCP(0x10d89cd0, 0x112d6f44, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:254 +0x47
net.(*TCPListener).Accept(0x10d89cd0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:264 +0x39
net/http.(*Server).Serve(0x112e3770, 0x31292680, 0x10d89cd0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:2117 +0x100
net/http/httptest.(*Server).goServe.func1(0x111374c0)
    c:/dev/go/src/net/http/httptest/server.go:237 +0x66
created by net/http/httptest.(*Server).goServe
    c:/dev/go/src/net/http/httptest/server.go:238 +0x50

goroutine 5951 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e21378, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e21340, 0x10d00000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa1a0, 0x10d00000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10cfa1a0, 0x10e0d924, 0x10d00000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0af0, 0x10d00000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17dd0)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17dd0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d8f0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5527 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112d80)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5532 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112f00)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5764 [IO wait]:
net.runtime_pollWait(0x31292758, 0x72, 0x10ca0284)
    c:/dev/go/src/runtime/netpoll.go:160 +0x55
net.(*pollDesc).Wait(0x10d66fb0, 0x72, 0x0, 0x0)
    c:/dev/go/src/net/fd_poll_runtime.go:73 +0x35
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10d66f18, 0x847468, 0x8, 0x10ca02a0, 0x10e208c0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:183 +0x122
net.(*netFD).acceptOne(0x10d66ee0, 0x10e207e0, 0x2, 0x2, 0x10d66f18, 0x10e3bf0c, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:583 +0x1d4
net.(*netFD).accept(0x10d66ee0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:613 +0x11f
net.(*TCPListener).AcceptTCP(0x10d894d0, 0x10e3bf44, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:254 +0x47
net.(*TCPListener).Accept(0x10d894d0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/tcpsock_posix.go:264 +0x39
net/http.(*Server).Serve(0x112e2eb0, 0x31292680, 0x10d894d0, 0x0, 0x0)
    c:/dev/go/src/net/http/server.go:2117 +0x100
net/http/httptest.(*Server).goServe.func1(0x112e5300)
    c:/dev/go/src/net/http/httptest/server.go:237 +0x66
created by net/http/httptest.(*Server).goServe
    c:/dev/go/src/net/http/httptest/server.go:238 +0x50

goroutine 5898 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8fdf8, 0x82c470, 0x7, 0x8fc0e4, 0x20, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10c8fdc0, 0x111c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10d89db0, 0x111c0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10d89db0, 0x10d06f84, 0x111c0000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca07a0, 0x111c0000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17b00)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17b00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06f50)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5530 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112ea0)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5939 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e20ff8, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e20fc0, 0x10cf6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10d89f60, 0x10cf6000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10d89f60, 0x10e0d5a4, 0x10cf6000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0910, 0x10cf6000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17c20)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17c20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d570)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5400 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112360)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5915 [chan receive]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8f998, 0x82c470, 0x7, 0x8fc0e4, 0x100, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:210 +0x422
net.(*netFD).Read(0x10c8f960, 0x112f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10dd88a8, 0x112f0000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10dd88a8, 0x10d06b24, 0x112f0000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10c88770, 0x112f0000, 0x1000, 0x1000, 0x2d0250, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x11241e00)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x11241e00, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06af0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5524 [chan receive]:
testing.(*T).Parallel(0x11112d20)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112d20)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportResponseHeaderTimeout(0x11112d20)
    c:/dev/go/src/net/http/transport_test.go:1393 +0xc8
testing.tRunner(0x11112d20, 0xa4ac8c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5402 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x111123c0)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5954 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e21458, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e21420, 0x10dac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa230, 0x10dac000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10cfa230, 0x10e0da04, 0x10dac000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0b60, 0x10dac000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17e30)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17e30, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d9d0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5679 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x10c48d80)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5531 [chan receive]:
testing.(*T).Parallel(0x11112f00)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112f00)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestCancelRequestWithChannelBeforeDo(0x11112f00)
    c:/dev/go/src/net/http/transport_test.go:1633 +0x39
testing.tRunner(0x11112f00, 0xa4acbc)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5525 [chan receive]:
testing.RunTests.func1(0x10c068c0, 0x11112d20)
    c:/dev/go/src/testing/testing.go:586 +0x3e
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:587 +0x76e

goroutine 5526 [chan receive]:
testing.(*T).Parallel(0x11112d80)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x11112d80)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportCancelRequest(0x11112d80)
    c:/dev/go/src/net/http/transport_test.go:1465 +0x68
testing.tRunner(0x11112d80, 0xa4ac98)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5924 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8fc38, 0x82c470, 0x7, 0x8fc0e4, 0x100, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10c8fc00, 0x111bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10dd8a58, 0x111bc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10dd8a58, 0x10d06dc4, 0x111bc000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10c888e0, 0x111bc000, 0x1000, 0x1000, 0x2d0250, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x11241f20)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x11241f20, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06d90)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5945 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e211b8, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e21180, 0x10cfc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10cfa080, 0x10cfc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10cfa080, 0x10e0d764, 0x10cfc000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0a00, 0x10cfc000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17d10)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17d10, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d730)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5904 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e20f18, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e20ee0, 0x10cf2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10d89ed0, 0x10cf2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10d89ed0, 0x10e0d4c4, 0x10cf2000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0890, 0x10cf2000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17bc0)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17bc0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d490)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5901 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10e20e38, 0x82c470, 0x7, 0x8fc0e4, 0x40, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10e20e00, 0x111c2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10d89e40, 0x111c2000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10d89e40, 0x10e0d3e4, 0x111c2000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10ca0820, 0x111c2000, 0x1000, 0x1000, 0x2d0000, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x10d17b60)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x10d17b60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10e0d3b0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

goroutine 5678 [chan receive]:
testing.(*T).Parallel(0x10c48d80)
    c:/dev/go/src/testing/testing.go:441 +0xf9
net/http_test.setParallel(0x10c48d80)
    c:/dev/go/src/net/http/main_test.go:90 +0x31
net/http_test.TestTransportClosesBodyOnError(0x10c48d80)
    c:/dev/go/src/net/http/transport_test.go:2509 +0x5e
testing.tRunner(0x10c48d80, 0xa4ad7c)
    c:/dev/go/src/testing/testing.go:473 +0x8f
created by testing.RunTests
    c:/dev/go/src/testing/testing.go:582 +0x6f2

goroutine 5918 [chan send]:
net.(*ioSrv).ExecIO(0x10c1ab30, 0x10c8fa78, 0x82c470, 0x7, 0x8fc0e4, 0x100, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:209 +0x403
net.(*netFD).Read(0x10c8fa40, 0x111b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/fd_windows.go:482 +0x129
net.(*conn).Read(0x10dd8938, 0x111b8000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
    c:/dev/go/src/net/net.go:172 +0xb9
net/http.noteEOFReader.Read(0x36ecc0, 0x10dd8938, 0x10d06c04, 0x111b8000, 0x1000, 0x1000, 0x2, 0x0, 0x0)
    c:/dev/go/src/net/http/transport.go:1687 +0x55
net/http.(*noteEOFReader).Read(0x10c887f0, 0x111b8000, 0x1000, 0x1000, 0x2d0250, 0x0, 0x0)
    <autogenerated>:284 +0xae
bufio.(*Reader).fill(0x11241e60)
    c:/dev/go/src/bufio/bufio.go:97 +0x172
bufio.(*Reader).Peek(0x11241e60, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    c:/dev/go/src/bufio/bufio.go:132 +0xad
net/http.(*persistConn).readLoop(0x10d06bd0)
    c:/dev/go/src/net/http/transport.go:1073 +0x13e
created by net/http.(*Transport).dialConn
    c:/dev/go/src/net/http/transport.go:857 +0xe21

--- FAIL: TestTransportPersistConnLeak (0.28s)
    transport_test.go:1044: goroutine growth: 21 -> 121 -> 35 (delta: 14)
    transport_test.go:1045: too many new goroutines
FAIL
exit status 1
FAIL    net/http    18.219s

C:\dev\go\src\net\http>

Am I seeing some unclosed network connections? Brad let me know, if you want me to try something.

Alex

@bradfitz
Copy link
Contributor

That's too hard to read. I only want to see the new goroutines.

See the leak check stuff I added in grpc: https://github.com/grpc/grpc-go/blob/master/test/end2end_test.go

// interestingGoroutines returns all goroutines we care about for the purpose
// of leak checking. It excludes testing or runtime ones.
func interestingGoroutines() (gs []string) {
    buf := make([]byte, 2<<20)
    buf = buf[:runtime.Stack(buf, true)]
    for _, g := range strings.Split(string(buf), "\n\n") {
        sl := strings.SplitN(g, "\n", 2)
        if len(sl) != 2 {
            continue
        }
        stack := strings.TrimSpace(sl[1])
        if strings.HasPrefix(stack, "testing.RunTests") {
            continue
        }

        if stack == "" ||
            strings.Contains(stack, "testing.Main(") ||
            strings.Contains(stack, "runtime.goexit") ||
            strings.Contains(stack, "created by runtime.gc") ||
            strings.Contains(stack, "interestingGoroutines") ||
            strings.Contains(stack, "runtime.MHeap_Scavenger") {
            continue
        }
        gs = append(gs, g)
    }
    sort.Strings(gs)
    return
}

// leakCheck snapshots the currently-running goroutines and returns a
// function to be run at the end of tests to see whether any
// goroutines leaked.
func leakCheck(t testing.TB) func() {
    orig := map[string]bool{}
    for _, g := range interestingGoroutines() {
        orig[g] = true
    }
    return func() {
        // Loop, waiting for goroutines to shut down.
        // Wait up to 5 seconds, but finish as quickly as possible.
        deadline := time.Now().Add(5 * time.Second)
        for {
            var leaked []string
            for _, g := range interestingGoroutines() {
                if !orig[g] {
                    leaked = append(leaked, g)
                }
            }
            if len(leaked) == 0 {
                return
            }
            if time.Now().Before(deadline) {
                time.Sleep(50 * time.Millisecond)
                continue
            }
            for _, g := range leaked {
                t.Errorf("Leaked goroutine: %v", g)
            }
            return
        }
    }
}

@alexbrainman
Copy link
Member

Sorry for late reply (I lost your reply). I have added leakCheck, but the package already has interestingGoroutines. The TestTransportPersistConnLeak know fails every time - I am sure that is not what you want. Please, let me know what to do. Thank you.

C:\dev\go\src\net\http>git diff
diff --git a/src/net/http/transport_test.go b/src/net/http/transport_test.go
index 63fa7ce..e36f645 100644
--- a/src/net/http/transport_test.go
+++ b/src/net/http/transport_test.go
@@ -979,8 +979,43 @@ func waitNumGoroutine(nmax int) int {
    return nfinal
 }

+// leakCheck snapshots the currently-running goroutines and returns a
+// function to be run at the end of tests to see whether any
+// goroutines leaked.
+func leakCheck(t testing.TB) func() {
+   orig := map[string]bool{}
+   for _, g := range interestingGoroutines() {
+       orig[g] = true
+   }
+   return func() {
+       // Loop, waiting for goroutines to shut down.
+       // Wait up to 5 seconds, but finish as quickly as possible.
+       deadline := time.Now().Add(5 * time.Second)
+       for {
+           var leaked []string
+           for _, g := range interestingGoroutines() {
+               if !orig[g] {
+                   leaked = append(leaked, g)
+               }
+           }
+           if len(leaked) == 0 {
+               return
+           }
+           if time.Now().Before(deadline) {
+               time.Sleep(50 * time.Millisecond)
+               continue
+           }
+           for _, g := range leaked {
+               t.Errorf("Leaked goroutine: %v", g)
+           }
+           return
+       }
+   }
+}
+
 // tests that persistent goroutine connections shut down when no longer desired.
 func TestTransportPersistConnLeak(t *testing.T) {
+   defer leakCheck(t)()
    setParallel(t)
    defer afterTest(t)
    gotReqCh := make(chan bool)

C:\dev\go\src\net\http>go test -short
--- FAIL: TestTransportPersistConnLeak (5.16s)
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10b5c480)
            c:/dev/go/src/testing/testing.go:530 +0x1d6
        testing.tRunner(0x10b5c480, 0x10c56ecc)
            c:/dev/go/src/testing/testing.go:549 +0x99
        testing.RunTests(0x802400, 0x9259a0, 0x158, 0x158, 0x40da01)
            c:/dev/go/src/testing/testing.go:736 +0x358
        testing.(*M).Run(0x10c56f84, 0x340018)
            c:/dev/go/src/testing/testing.go:672 +0x79
        net/http_test.TestMain(0x10c56f84)
            c:/dev/go/src/net/http/main_test.go:22 +0x21
        main.main()
            net/http/_test/_testmain.go:784 +0x100
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10bc8000)
            c:/dev/go/src/testing/testing.go:543 +0x232
        testing.tRunner(0x10bc8000, 0x802e00)
            c:/dev/go/src/testing/testing.go:549 +0x99
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10ed2480)
            c:/dev/go/src/testing/testing.go:543 +0x232
        runtime.Goexit()
            c:/dev/go/src/runtime/panic.go:340 +0xde
        testing.(*common).SkipNow(0x10ed2480)
            c:/dev/go/src/testing/testing.go:459 +0x32
        testing.(*common).Skip(0x10ed2480, 0x10bcfe48, 0x1, 0x1)
            c:/dev/go/src/testing/testing.go:442 +0x61
        net/http_test.TestTransportResponseHeaderTimeout(0x10ed2480)
            c:/dev/go/src/net/http/transport_test.go:1425 +0x20b
        testing.tRunner(0x10ed2480, 0x802e30)
            c:/dev/go/src/testing/testing.go:547 +0x8c
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10ed2500)
            c:/dev/go/src/testing/testing.go:543 +0x232
        runtime.Goexit()
            c:/dev/go/src/runtime/panic.go:340 +0xde
        testing.(*common).SkipNow(0x10ed2500)
            c:/dev/go/src/testing/testing.go:459 +0x32
        testing.(*common).Skip(0x10ed2500, 0x10bcdf48, 0x1, 0x1)
            c:/dev/go/src/testing/testing.go:442 +0x61
        net/http_test.TestTransportCancelRequest(0x10ed2500)
            c:/dev/go/src/net/http/transport_test.go:1497 +0x1ab
        testing.tRunner(0x10ed2500, 0x802d38)
            c:/dev/go/src/testing/testing.go:547 +0x8c
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10ed2600)
            c:/dev/go/src/testing/testing.go:543 +0x232
        runtime.Goexit()
            c:/dev/go/src/runtime/panic.go:340 +0xde
        testing.(*common).SkipNow(0x10ed2600)
            c:/dev/go/src/testing/testing.go:459 +0x32
        testing.(*common).Skip(0x10ed2600, 0x10bd2f48, 0x1, 0x1)
            c:/dev/go/src/testing/testing.go:442 +0x61
        net/http_test.TestCancelRequestWithChannel(0x10ed2600)
            c:/dev/go/src/net/http/transport_test.go:1607 +0x1ab
        testing.tRunner(0x10ed2600, 0x8027b4)
            c:/dev/go/src/testing/testing.go:547 +0x8c
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10ed2680)
            c:/dev/go/src/testing/testing.go:543 +0x232
        testing.tRunner(0x10ed2680, 0x8027b0)
            c:/dev/go/src/testing/testing.go:549 +0x99
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
    transport_test.go:1009: Leaked goroutine: testing.tRunner.func1(0x10ed3280)
            c:/dev/go/src/testing/testing.go:543 +0x232
        testing.tRunner(0x10ed3280, 0x802d48)
            c:/dev/go/src/testing/testing.go:549 +0x99
        created by testing.(*T).run
            c:/dev/go/src/testing/testing.go:578 +0x2b8
FAIL
exit status 1
FAIL    net/http    21.672s

C:\dev\go\src\net\http>

Alex

@bradfitz bradfitz modified the milestones: Go1.6.1, Go1.6.2 Apr 7, 2016
@rsc rsc modified the milestones: Go1.7, Go1.6.2 Apr 7, 2016
@bradfitz bradfitz added the Testing An issue that has been verified to require only test changes, not just a test failure. label May 18, 2016
@bradfitz
Copy link
Contributor

Looks like this was already fixed in 34f0c0b for #14887

@golang golang locked and limited conversation to collaborators May 20, 2017
@golang golang unlocked this conversation Aug 12, 2019
@golang golang locked as resolved and limited conversation to collaborators Aug 12, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge Testing An issue that has been verified to require only test changes, not just a test failure.
Projects
None yet
Development

No branches or pull requests

6 participants