Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: application hangs after a while #28441

Closed
dannin opened this issue Oct 27, 2018 · 15 comments
Closed

net/http: application hangs after a while #28441

dannin opened this issue Oct 27, 2018 · 15 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@dannin
Copy link

dannin commented Oct 27, 2018

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

1.10.1

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

linux arm
1cpu

What did you do?

Application is left on with repeated http requests

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

Application not to hang

What did you see instead?

Application becomes unresponsive

@dannin
Copy link
Author

dannin commented Oct 27, 2018

SIGABRT: abort
PC=0x6bb00 m=0 sigcode=0

goroutine 0 [idle]:
runtime.futex(0xf0a7d0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xf0a7d0, 0x0, 0x0, 0x413d0, ...)
/usr/local/go/src/runtime/sys_linux_arm.s:266 +0x1c
runtime.futexsleep(0xf0a7d0, 0x0, 0xffffffff, 0xffffffff)
/usr/local/go/src/runtime/os_linux.go:45 +0x74
runtime.notesleep(0xf0a7d0)
/usr/local/go/src/runtime/lock_futex.go:151 +0xac
runtime.stopm()
/usr/local/go/src/runtime/proc.go:1952 +0xac
runtime.exitsyscall0(0x11fa50a0)
/usr/local/go/src/runtime/proc.go:3129 +0x108
runtime.mcall(0x0)
/usr/local/go/src/runtime/asm_arm.s:285 +0x5c

goroutine 1 [runnable]:
internal/poll.runtime_pollWait(0x76f0ee40, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5d784, 0x72, 0xa3c00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5d784, 0xffffff00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Accept(0x11c5d770, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:372 +0x174
net.(*netFD).accept(0x11c5d770, 0x11f0f0c0, 0x11c1c03c, 0x11c1c03c)
/usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0x11dc1ac8, 0x0, 0x1, 0x24b144)
/usr/local/go/src/net/tcpsock_posix.go:136 +0x20
net.(*TCPListener).AcceptTCP(0x11dc1ac8, 0x1238ff60, 0x12b78, 0x112ec)
/usr/local/go/src/net/tcpsock.go:246 +0x3c
net/http.tcpKeepAliveListener.Accept(0x11dc1ac8, 0x44674, 0x24a8a4, 0x120e3ed8, 0x120e3ef8)
/usr/local/go/src/net/http/server.go:3216 +0x1c
crypto/tls.(*listener).Accept(0x11f034c0, 0x7adbe8, 0x1238ff20, 0x841520, 0x124807e0)
/usr/local/go/src/crypto/tls/tls.go:52 +0x28
net/http.(*Server).Serve(0x12425a00, 0x8403a0, 0x11f034c0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2770 +0x150
net/http.(*Server).ServeTLS(0x12425a00, 0x841220, 0x11dc1ac8, 0x6e655c, 0x13, 0x6e5655, 0x12, 0x0, 0x6bf7f8)
/usr/local/go/src/net/http/server.go:2839 +0x1bc
net/http.(*Server).ListenAndServeTLS(0x12425a00, 0x6e655c, 0x13, 0x6e5655, 0x12, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:3032 +0xe4
gitsite/store/tools/gitapp/server.Start(0x11dc63a0, 0x19, 0x11f0fe00, 0x1, 0xf39b2000, 0x105e)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/server/server.go:118 +0x358
main.main()
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/main.go:212 +0x828

goroutine 5 [select, 1969 minutes]:
gitsite/store/tools/vendor/github.com/user/glog.(*loggingT).flushDaemon(0xf0a1c0)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/user/glog/glog.go:949 +0x154
created by gitsite/store/tools/vendor/github.com/user/glog.init.0
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/user/glog/glog.go:414 +0x210

goroutine 9 [chan receive, 2046 minutes]:
gitsite/store/tools/myproductapp/site/datastorage.(*Store).persistLoop(0x11cd3050)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:170 +0x30
created by gitsite/store/tools/myproductapp/site/datastorage.New
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:81 +0xd0

goroutine 10 [chan receive, 2046 minutes]:
gitsite/store/tools/myproductapp/site/datastorage.(*Store).persistLoop(0x11cd2000)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:170 +0x30
created by gitsite/store/tools/myproductapp/site/datastorage.New
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:81 +0xd0

goroutine 11 [chan receive, 2046 minutes]:
gitsite/store/tools/myproductapp/site/datastorage.(*Store).persistLoop(0x11cd2600)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:170 +0x30
created by gitsite/store/tools/myproductapp/site/datastorage.New
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/myproductapp/site/datastorage/datastorage.go:81 +0xd0

goroutine 12 [runnable]:
internal/poll.runtime_pollWait(0x76f0efc0, 0x72, 0x11ec90f0)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5c474, 0x72, 0x1229d300, 0x1000, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5c474, 0x11ec9000, 0x10, 0x10)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).ReadMsg(0x11c5c460, 0x11ec90f0, 0x10, 0x10, 0x1229d310, 0x1000, 0x1000, 0x0, 0x0, 0x0, ...)
/usr/local/go/src/internal/poll/fd_unix.go:231 +0x168
net.(*netFD).readMsg(0x11c5c460, 0x11ec90f0, 0x10, 0x10, 0x1229d310, 0x1000, 0x1000, 0x2a3548, 0x125b7e4c, 0x11e38ab0, ...)
/usr/local/go/src/net/fd_unix.go:214 +0x50
net.(*UnixConn).readMsg(0x11d6d6c0, 0x11ec90f0, 0x10, 0x10, 0x1229d310, 0x1000, 0x1000, 0x12161500, 0x11e38903, 0x0, ...)
/usr/local/go/src/net/unixsock_posix.go:115 +0x50
net.(*UnixConn).ReadMsgUnix(0x11d6d6c0, 0x11ec90f0, 0x10, 0x10, 0x1229d310, 0x1000, 0x1000, 0x7, 0x11e38abe, 0x837092, ...)
/usr/local/go/src/net/unixsock.go:137 +0x70
gitsite/store/tools/vendor/github.com/godbus/dbus.(*oobReader).Read(0x1229d300, 0x11ec90f0, 0x10, 0x10, 0x7fffffff, 0x1229d300, 0x0)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/godbus/dbus/transport_unix.go:21 +0x54
io.ReadAtLeast(0x83df08, 0x1229d300, 0x11ec90f0, 0x10, 0x10, 0x10, 0x66be10, 0x14ff01, 0x1229d300)
/usr/local/go/src/io/io.go:309 +0x6c
io.ReadFull(0x83df08, 0x1229d300, 0x11ec90f0, 0x10, 0x10, 0x11f9fbc0, 0x125b7ef0, 0x11f9fb01)
/usr/local/go/src/io/io.go:327 +0x40
gitsite/store/tools/vendor/github.com/godbus/dbus.(*unixTransport).ReadMessage(0x11d6d6a0, 0xb7, 0xb7, 0x12149e01)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/godbus/dbus/transport_unix.go:85 +0xb8
gitsite/store/tools/vendor/github.com/godbus/dbus.(*Conn).inWorker(0x11c5c500)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/godbus/dbus/conn.go:258 +0x28
created by gitsite/store/tools/vendor/github.com/godbus/dbus.(*Conn).Auth
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/vendor/github.com/godbus/dbus/auth.go:118 +0x4d8

goroutine 21 [select, 1971 minutes]:
gitsite/store/tools/utils/installer.(*App).myproductappRestarterLoop(0x11cc5800)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:456 +0xcc
created by gitsite/store/tools/utils/installer.NewApp
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:727 +0x8fc

goroutine 16 [chan receive, 2044 minutes]:
gitsite/store/tools/connman.(*Signaler).signalListener(0x11db4f30, 0x11c5c500)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/connman/connman_signal.go:37 +0x58
created by gitsite/store/tools/connman.NewSignaler
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/connman/connman_signal.go:29 +0x88

goroutine 23 [IO wait, 1976 minutes]:
internal/poll.runtime_pollWait(0x76f0ef40, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5d3c4, 0x72, 0xa3c00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5d3c4, 0xffffff00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Accept(0x11c5d3b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:372 +0x174
net.(*netFD).accept(0x11c5d3b0, 0x11f47000, 0x1000, 0x1000)
/usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0x11d28ea0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/tcpsock_posix.go:136 +0x20
net.(*TCPListener).Accept(0x11d28ea0, 0x842bb8, 0x11e0ed20, 0x0, 0x0)
/usr/local/go/src/net/tcpsock.go:259 +0x3c
gitsite/store/tools/gitapp/update.(*Updater).startStatusServer.func1(0x8403c0, 0x11d28ea0, 0x11cc2e70)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/update/update.go:266 +0x24
created by gitsite/store/tools/gitapp/update.(*Updater).startStatusServer
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/update/update.go:264 +0x1a0

goroutine 24 [sleep, 1976 minutes]:
time.Sleep(0xb2c97000, 0x8b)
/usr/local/go/src/runtime/time.go:102 +0x154
gitsite/store/tools/gitapp/update.(*Updater).AutoUpdater(0x11cc2e70)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/update/update.go:767 +0x24
created by gitsite/store/tools/gitapp/update.NewUpdater
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/update/update.go:157 +0x274

goroutine 25 [select, 1969 minutes]:
gitsite/store/tools/datapump.(*Logger).notifyIncomingItemLoop(0x11d8c500, 0x11d6a450, 0x11e237c0, 0x11e23840)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:389 +0x168
gitsite/store/tools/datapump.(*Logger).Start.func1(0x11d8c500, 0x11d6a450)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:263 +0x34
created by gitsite/store/tools/datapump.(*Logger).Start
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:262 +0x194

goroutine 26 [select, 1969 minutes]:
gitsite/store/tools/datapump.(*Logger).notifySendNextItemLoop(0x11d8c500, 0x11d6a450, 0x11e23780, 0x11e23840)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:541 +0x17c
gitsite/store/tools/datapump.(*Logger).Start.func2(0x11d8c500, 0x11d6a450)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:266 +0x34
created by gitsite/store/tools/datapump.(*Logger).Start
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:265 +0x1b8

goroutine 27 [select, 1971 minutes]:
gitsite/store/tools/datapump.(*Logger).processBatchFilesLoop(0x11d8c500, 0x11d6a450, 0x11e23840)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:1143 +0xe8
gitsite/store/tools/datapump.(*Logger).Start.func3(0x11d8c500, 0x11d6a450)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:272 +0x2c
created by gitsite/store/tools/datapump.(*Logger).Start
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/datapump/datapump.go:271 +0x1f4

goroutine 330 [select]:
net/http.(*http2serverConn).serve(0x11c60900)
/usr/local/go/src/net/http/h2_bundle.go:4505 +0x48c
net/http.(*http2Server).ServeConn(0x11ed73e0, 0x842b58, 0x11c79680, 0x125bbdf8)
/usr/local/go/src/net/http/h2_bundle.go:4124 +0x5d4
net/http.http2utilsureServer.func1(0x12425a00, 0x11c79680, 0x83e688, 0x11d4acb0)
/usr/local/go/src/net/http/h2_bundle.go:3962 +0x64
net/http.(*conn).serve(0x1238ff20, 0x841520, 0x11f6a340)
/usr/local/go/src/net/http/server.go:1751 +0xce8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x208

goroutine 29 [chan receive, 1985 minutes]:
gitsite/store/tools/utils/installer.(*App).syncAllLoop(0x11cc5800)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:339 +0x70
created by gitsite/store/tools/utils/installer.NewApp
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:745 +0xaec

goroutine 220 [runnable]:
internal/poll.runtime_pollWait(0x76f0edc0, 0x72, 0x0)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5d8c4, 0x72, 0xa3c00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5d8c4, 0xffffff00, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Accept(0x11c5d8b0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:372 +0x174
net.(*netFD).accept(0x11c5d8b0, 0x11c42704, 0x11c2aa00, 0x66980)
/usr/local/go/src/net/fd_unix.go:238 +0x20
net.(*TCPListener).accept(0x11dc1bb0, 0x124676b4, 0x150bc4, 0x124676b4)
/usr/local/go/src/net/tcpsock_posix.go:136 +0x20
net.(*TCPListener).AcceptTCP(0x11dc1bb0, 0x124676b4, 0x12425a80, 0x124676a0)
/usr/local/go/src/net/tcpsock.go:246 +0x3c
net/http.tcpKeepAliveListener.Accept(0x11dc1bb0, 0x11c16040, 0x6360c0, 0xf01cd0, 0x6c5230)
/usr/local/go/src/net/http/server.go:3216 +0x1c
net/http.(*Server).Serve(0x12425a80, 0x841220, 0x11dc1bb0, 0x0, 0x0)
/usr/local/go/src/net/http/server.go:2770 +0x150
net/http.(*Server).ListenAndServe(0x12425a80, 0x12467420, 0x0)
/usr/local/go/src/net/http/server.go:2711 +0x84
gitsite/store/tools/gitapp/server.Start.func2(0x12467420)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/server/server.go:105 +0x88
created by gitsite/store/tools/gitapp/server.Start
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/gitapp/server/server.go:98 +0x3fc

goroutine 213 [sleep, 1975 minutes]:
time.Sleep(0xb2c97000, 0x8b)
/usr/local/go/src/runtime/time.go:102 +0x154
gitsite/store/tools/utils/installer.NewApp.func2(0x11d6c188)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:809 +0x30
created by gitsite/store/tools/utils/installer.NewApp
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:783 +0xcbc

goroutine 221 [select]:
net/http.(*http2serverConn).serve(0x11c60240)
/usr/local/go/src/net/http/h2_bundle.go:4505 +0x48c
net/http.(*http2Server).ServeConn(0x11ed73e0, 0x842b58, 0x11c78480, 0x11d0ddf8)
/usr/local/go/src/net/http/h2_bundle.go:4124 +0x5d4
net/http.http2utilsureServer.func1(0x12425a00, 0x11c78480, 0x83e688, 0x11d6c110)
/usr/local/go/src/net/http/h2_bundle.go:3962 +0x64
net/http.(*conn).serve(0x1238e000, 0x841520, 0x11f9eb00)
/usr/local/go/src/net/http/server.go:1751 +0xce8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x208

goroutine 219 [select, 1969 minutes]:
gitsite/store/tools/utils/installer.ConnmanManagerLoop(0xf8475800, 0xd, 0xf39b2000, 0x105e, 0x11cc5800)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/connmanManager.go:340 +0x2dc
created by main.main
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/main.go:207 +0x558

goroutine 215 [sleep, 1985 minutes]:
time.Sleep(0x30b8a000, 0x346)
/usr/local/go/src/runtime/time.go:102 +0x154
gitsite/store/tools/utils/installer.NewApp.func3(0x11d6c188)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:822 +0x24
created by gitsite/store/tools/utils/installer.NewApp
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/app.go:820 +0xd34

goroutine 218 [sleep, 1985 minutes]:
time.Sleep(0x30b8a000, 0x346)
/usr/local/go/src/runtime/time.go:102 +0x154
main.main.func2(0x11cc5800)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/main.go:202 +0x28
created by main.main
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/main.go:188 +0x4d8

goroutine 222 [select]:
net/http.(*http2serverConn).serve(0x11c60360)
/usr/local/go/src/net/http/h2_bundle.go:4505 +0x48c
net/http.(*http2Server).ServeConn(0x11ed73e0, 0x842b58, 0x11c786c0, 0x11c3bdf8)
/usr/local/go/src/net/http/h2_bundle.go:4124 +0x5d4
net/http.http2utilsureServer.func1(0x12425a00, 0x11c786c0, 0x83e688, 0x11d6c6b8)
/usr/local/go/src/net/http/h2_bundle.go:3962 +0x64
net/http.(*conn).serve(0x1238e0c0, 0x841520, 0x11f9ec80)
/usr/local/go/src/net/http/server.go:1751 +0xce8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x208

goroutine 225 [runnable]:
internal/poll.runtime_pollWait(0x76f0eec0, 0x72, 0x11e66800)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5dc84, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5dc84, 0x11e66800, 0x400, 0x400)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x11c5dc70, 0x11e66800, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x11c5dc70, 0x11e66800, 0x400, 0x400, 0x11e8aa00, 0x11c62960, 0xa0)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11d6c000, 0x11e66800, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
crypto/tls.(*block).readFromUntil(0x11f9eb40, 0x76f1d1b8, 0x11d6c000, 0x5, 0x11d6c000, 0x11d52f00)
/usr/local/go/src/crypto/tls/conn.go:493 +0x78
crypto/tls.(*Conn).readRecord(0x11c78480, 0x7ae517, 0x11c78520, 0x1)
/usr/local/go/src/crypto/tls/conn.go:595 +0x9c
crypto/tls.(*Conn).Read(0x11c78480, 0x123aa120, 0x9, 0x9, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1156 +0xcc
io.ReadAtLeast(0x76f1d2c0, 0x11c78480, 0x123aa120, 0x9, 0x9, 0x9, 0x11e52148, 0x11e8c320, 0x0)
/usr/local/go/src/io/io.go:309 +0x6c
io.ReadFull(0x76f1d2c0, 0x11c78480, 0x123aa120, 0x9, 0x9, 0x11e271c0, 0x1211df58, 0x7f28ca01)
/usr/local/go/src/io/io.go:327 +0x40
net/http.http2readFrameHeader(0x123aa120, 0x9, 0x9, 0x76f1d2c0, 0x11c78480, 0x0, 0x0, 0x0, 0x11fabaa4, 0x11e271c8)
/usr/local/go/src/net/http/h2_bundle.go:1517 +0x50
net/http.(*http2Framer).ReadFrame(0x123aa100, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1775 +0x78
net/http.(*http2serverConn).readFrames(0x11c60240)
/usr/local/go/src/net/http/h2_bundle.go:4392 +0x78
created by net/http.(*http2serverConn).serve
/usr/local/go/src/net/http/h2_bundle.go:4497 +0x2a0

goroutine 329 [select]:
net/http.(*http2serverConn).serve(0x11c60a20)
/usr/local/go/src/net/http/h2_bundle.go:4505 +0x48c
net/http.(*http2Server).ServeConn(0x11ed73e0, 0x842b58, 0x11c79440, 0x125b9df8)
/usr/local/go/src/net/http/h2_bundle.go:4124 +0x5d4
net/http.http2utilsureServer.func1(0x12425a00, 0x11c79440, 0x83e688, 0x11d4b490)
/usr/local/go/src/net/http/h2_bundle.go:3962 +0x64
net/http.(*conn).serve(0x1238fe60, 0x841520, 0x11f6a240)
/usr/local/go/src/net/http/server.go:1751 +0xce8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x208

goroutine 354 [runnable]:
internal/poll.runtime_pollWait(0x76f0eb40, 0x72, 0x11fe8c00)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x1239fb94, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x1239fb94, 0x11fe8c00, 0x400, 0x400)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x1239fb80, 0x11fe8c00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x1239fb80, 0x11fe8c00, 0x400, 0x400, 0x11d4ca00, 0x12149cc0, 0xa0)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11d4ac10, 0x11fe8c00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
crypto/tls.(*block).readFromUntil(0x11f6a280, 0x76f1d1b8, 0x11d4ac10, 0x5, 0x11d4ac10, 0x11d52f00)
/usr/local/go/src/crypto/tls/conn.go:493 +0x78
crypto/tls.(*Conn).readRecord(0x11c79440, 0x7ae517, 0x11c794e0, 0x1)
/usr/local/go/src/crypto/tls/conn.go:595 +0x9c
crypto/tls.(*Conn).Read(0x11c79440, 0x123724a0, 0x9, 0x9, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1156 +0xcc
io.ReadAtLeast(0x76f1d2c0, 0x11c79440, 0x123724a0, 0x9, 0x9, 0x9, 0x11ec8d58, 0x11e8c140, 0x0)
/usr/local/go/src/io/io.go:309 +0x6c
io.ReadFull(0x76f1d2c0, 0x11c79440, 0x123724a0, 0x9, 0x9, 0x11f77cc0, 0x1238af58, 0x7f28ca01)
/usr/local/go/src/io/io.go:327 +0x40
net/http.http2readFrameHeader(0x123724a0, 0x9, 0x9, 0x76f1d2c0, 0x11c79440, 0x0, 0x0, 0x0, 0x11f0c9a4, 0x11f77cc8)
/usr/local/go/src/net/http/h2_bundle.go:1517 +0x50
net/http.(*http2Framer).ReadFrame(0x12372480, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1775 +0x78
net/http.(*http2serverConn).readFrames(0x11c60a20)
/usr/local/go/src/net/http/h2_bundle.go:4392 +0x78
created by net/http.(*http2serverConn).serve
/usr/local/go/src/net/http/h2_bundle.go:4497 +0x2a0

goroutine 318914 [runnable]:
net/http.(*http2serverConn).runHandler(0x11c60900, 0x11e8cef8, 0x12373100, 0x11f7a6c0)
/usr/local/go/src/net/http/h2_bundle.go:5739
created by net/http.(*http2serverConn).processHeaders
/usr/local/go/src/net/http/h2_bundle.go:5494 +0x3a4

goroutine 318899 [runnable]:
internal/poll.runtime_pollWait(0x76f0e840, 0x72, 0x11ecc000)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x12162654, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x12162654, 0x11ecc000, 0x1000, 0x1000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x12162640, 0x11ecc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x12162640, 0x11ecc000, 0x1000, 0x1000, 0x651b88, 0x1641d8, 0x20394)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11e8cab0, 0x11ecc000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
net/http.(*persistConn).Read(0x11c62dc0, 0x11ecc000, 0x1000, 0x1000, 0x11e52410, 0x2581a4, 0x40)
/usr/local/go/src/net/http/transport.go:1453 +0x174
bufio.(*Reader).fill(0x11fa2810)
/usr/local/go/src/bufio/bufio.go:100 +0x104
bufio.(*Reader).Peek(0x11fa2810, 0x1, 0x11ea48c0, 0x11c41604, 0x7adae4, 0x0, 0xa1b40)
/usr/local/go/src/bufio/bufio.go:132 +0x28
net/http.(*persistConn).readLoop(0x11c62dc0)
/usr/local/go/src/net/http/transport.go:1601 +0x14c
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1237 +0x7b0

goroutine 248 [runnable]:
internal/poll.runtime_pollWait(0x76f0ed40, 0x72, 0x1248ac00)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x11c5dcd4, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x11c5dcd4, 0x1248ac00, 0x400, 0x400)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x11c5dcc0, 0x1248ac00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x11c5dcc0, 0x1248ac00, 0x400, 0x400, 0x11e8a500, 0x11c625a0, 0xa0)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11d6c050, 0x1248ac00, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
crypto/tls.(*block).readFromUntil(0x11f9fec0, 0x76f1d1b8, 0x11d6c050, 0x5, 0x11d6c050, 0x11d52f00)
/usr/local/go/src/crypto/tls/conn.go:493 +0x78
crypto/tls.(*Conn).readRecord(0x11c786c0, 0x7ae517, 0x11c78760, 0x1)
/usr/local/go/src/crypto/tls/conn.go:595 +0x9c
crypto/tls.(*Conn).Read(0x11c786c0, 0x123aa9a0, 0x9, 0x9, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1156 +0xcc
io.ReadAtLeast(0x76f1d2c0, 0x11c786c0, 0x123aa9a0, 0x9, 0x9, 0x9, 0x11e520f8, 0x11e8c208, 0x0)
/usr/local/go/src/io/io.go:309 +0x6c
io.ReadFull(0x76f1d2c0, 0x11c786c0, 0x123aa9a0, 0x9, 0x9, 0x11f77840, 0x12006f58, 0x7f28ca01)
/usr/local/go/src/io/io.go:327 +0x40
net/http.http2readFrameHeader(0x123aa9a0, 0x9, 0x9, 0x76f1d2c0, 0x11c786c0, 0x0, 0x0, 0x0, 0x11d57fe4, 0x11f77848)
/usr/local/go/src/net/http/h2_bundle.go:1517 +0x50
net/http.(*http2Framer).ReadFrame(0x123aa980, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1775 +0x78
net/http.(*http2serverConn).readFrames(0x11c60360)
/usr/local/go/src/net/http/h2_bundle.go:4392 +0x78
created by net/http.(*http2serverConn).serve
/usr/local/go/src/net/http/h2_bundle.go:4497 +0x2a0

goroutine 333 [runnable]:
internal/poll.runtime_pollWait(0x76f0ecc0, 0x72, 0x11fe9400)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x1239fbe4, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x1239fbe4, 0x11fe9400, 0x400, 0x400)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x1239fbd0, 0x11fe9400, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x1239fbd0, 0x11fe9400, 0x400, 0x400, 0x11e8af00, 0x11c635e0, 0xa0)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11d4ac28, 0x11fe9400, 0x400, 0x400, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
crypto/tls.(*block).readFromUntil(0x11f6a380, 0x76f1d1b8, 0x11d4ac28, 0x5, 0x11d4ac28, 0x11d52f00)
/usr/local/go/src/crypto/tls/conn.go:493 +0x78
crypto/tls.(*Conn).readRecord(0x11c79680, 0x7ae517, 0x11c79720, 0x1)
/usr/local/go/src/crypto/tls/conn.go:595 +0x9c
crypto/tls.(*Conn).Read(0x11c79680, 0x123abda0, 0x9, 0x9, 0x0, 0x0, 0x0)
/usr/local/go/src/crypto/tls/conn.go:1156 +0xcc
io.ReadAtLeast(0x76f1d2c0, 0x11c79680, 0x123abda0, 0x9, 0x9, 0x9, 0x11e52b08, 0x11e8cf60, 0x0)
/usr/local/go/src/io/io.go:309 +0x6c
io.ReadFull(0x76f1d2c0, 0x11c79680, 0x123abda0, 0x9, 0x9, 0x11eaf7c0, 0x1211ef58, 0x7f28ca01)
/usr/local/go/src/io/io.go:327 +0x40
net/http.http2readFrameHeader(0x123abda0, 0x9, 0x9, 0x76f1d2c0, 0x11c79680, 0x0, 0x0, 0x0, 0x11f6f4e4, 0x11eaf7c8)
/usr/local/go/src/net/http/h2_bundle.go:1517 +0x50
net/http.(*http2Framer).ReadFrame(0x123abd80, 0x0, 0x0, 0x0, 0x0)
/usr/local/go/src/net/http/h2_bundle.go:1775 +0x78
net/http.(*http2serverConn).readFrames(0x11c60900)
/usr/local/go/src/net/http/h2_bundle.go:4392 +0x78
created by net/http.(*http2serverConn).serve
/usr/local/go/src/net/http/h2_bundle.go:4497 +0x2a0

goroutine 295 [running]:
goroutine running on other thread; stack unavailable
created by net/http.(*http2serverConn).serve
/usr/local/go/src/net/http/h2_bundle.go:4497 +0x2a0

goroutine 292 [select]:
net/http.(*http2serverConn).serve(0x11c60480)
/usr/local/go/src/net/http/h2_bundle.go:4505 +0x48c
net/http.(*http2Server).ServeConn(0x11ed73e0, 0x842b58, 0x11c78d80, 0x11d07df8)
/usr/local/go/src/net/http/h2_bundle.go:4124 +0x5d4
net/http.http2utilsureServer.func1(0x12425a00, 0x11c78d80, 0x83e688, 0x11d6d3f0)
/usr/local/go/src/net/http/h2_bundle.go:3962 +0x64
net/http.(*conn).serve(0x1238f680, 0x841520, 0x11dbc060)
/usr/local/go/src/net/http/server.go:1751 +0xce8
created by net/http.(*Server).Serve
/usr/local/go/src/net/http/server.go:2795 +0x208

goroutine 318916 [runnable]:
net/http.(*http2serverConn).runHandler(0x11c60900, 0x11e8cfb8, 0x12373300, 0x11f7a6f0)
/usr/local/go/src/net/http/h2_bundle.go:5739
created by net/http.(*http2serverConn).processHeaders
/usr/local/go/src/net/http/h2_bundle.go:5494 +0x3a4

goroutine 318915 [runnable]:
net/http.(*http2serverConn).runHandler(0x11c60900, 0x11e8cf50, 0x12373200, 0x11f7a6e0)
/usr/local/go/src/net/http/h2_bundle.go:5739
created by net/http.(*http2serverConn).processHeaders
/usr/local/go/src/net/http/h2_bundle.go:5494 +0x3a4

goroutine 318913 [runnable]:
syscall.readlen(0x12, 0x12120b00, 0x4, 0x4, 0x11d12560, 0x5)
/usr/local/go/src/syscall/zsyscall_linux_arm.go:1042 +0xf0
syscall.forkExec(0x11e52a08, 0x8, 0x11dd4d40, 0x3, 0x4, 0x12120bbc, 0x10, 0x0, 0x11e52a50)
/usr/local/go/src/syscall/exec_unix.go:203 +0x284
syscall.StartProcess(0x11e52a08, 0x8, 0x11dd4d40, 0x3, 0x4, 0x12120bbc, 0x2, 0x4, 0x11c1a000, 0x3)
/usr/local/go/src/syscall/exec_unix.go:241 +0x44
os.startProcess(0x11e52a08, 0x8, 0x11dd4d40, 0x3, 0x4, 0x12120c90, 0x11d122e0, 0x4, 0x4)
/usr/local/go/src/os/exec_posix.go:46 +0x148
os.StartProcess(0x11e52a08, 0x8, 0x11dd4d40, 0x3, 0x4, 0x12120c90, 0x0, 0x0, 0x8)
/usr/local/go/src/os/exec.go:102 +0x58
os/exec.(*Cmd).Start(0x11f44370, 0x1, 0x1243c960)
/usr/local/go/src/os/exec/exec.go:379 +0x390
os/exec.(*Cmd).Run(0x11f44370, 0x1243c960, 0x11f44370)
/usr/local/go/src/os/exec/exec.go:302 +0x1c
os/exec.(*Cmd).CombinedOutput(0x11f44370, 0x2, 0x12120d24, 0x2, 0x2, 0x11f44370)
/usr/local/go/src/os/exec/exec.go:521 +0xf8
gitsite/store/tools/utils/installer.cmdmyproductapp(0x6dd0da, 0x6, 0x0, 0x11c1c048, 0x3e75c8, 0x6dbf2e, 0x6dbf2e)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/myproductapp.go:50 +0x7c
gitsite/store/tools/utils/installer.(*App).myproductappStatus(0x11cc5800, 0x7ae028, 0x1233fdec, 0x840a80, 0x11f7a650, 0x6dbf2e)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/myproductapp.go:296 +0x30
gitsite/store/tools/utils/installer.(*App).myproductappStatusHandler(0x11cc5800, 0x83f5d0, 0x11e8cda8, 0x840a80, 0x11f7a650, 0x12373000, 0x0, 0x0)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/myproductapp.go:366 +0x5c
gitsite/store/tools/utils/installer.(*App).myproductappStatusHandler-fm(0x83f5d0, 0x11e8cda8, 0x840a80, 0x11f7a650, 0x12373000, 0x0, 0x0)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/utils/installer/routes.go:349 +0x44
gitsite/store/tools/restapi.Handle.func1(0x83f5d0, 0x11e8cda8, 0x840a80, 0x11f7a650, 0x12373000, 0x11f7a650, 0xa)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/restapi/chain_handler.go:68 +0x78
gitsite/store/tools/restapi.(*Router).ServeHTTP(0x11df2f00, 0x8404e0, 0x11e8cd80, 0x12373000)
/home/jenkins/workspace/myproduct/myproductapp/myproducttest/go/src/gitsite/store/tools/restapi/router.go:262 +0x560
net/http.(*ServeMux).ServeHTTP(0x11f0fe00, 0x8404e0, 0x11e8cd80, 0x12373000)
/usr/local/go/src/net/http/server.go:2337 +0x108
net/http.serverHandler.ServeHTTP(0x12425a00, 0x8404e0, 0x11e8cd80, 0x12373000)
/usr/local/go/src/net/http/server.go:2694 +0x74
net/http.initNPNRequest.ServeHTTP(0x11c79680, 0x12425a00, 0x8404e0, 0x11e8cd80, 0x12373000)
/usr/local/go/src/net/http/server.go:3260 +0x60
net/http.(Handler).ServeHTTP-fm(0x8404e0, 0x11e8cd80, 0x12373000)
/usr/local/go/src/net/http/h2_bundle.go:5475 +0x3c
net/http.(*http2serverConn).runHandler(0x11c60900, 0x11e8cd80, 0x12373000, 0x11f7a610)
/usr/local/go/src/net/http/h2_bundle.go:5760 +0x70
created by net/http.(*http2serverConn).processHeaders
/usr/local/go/src/net/http/h2_bundle.go:5494 +0x3a4

goroutine 318900 [select]:
net/http.(*persistConn).writeLoop(0x11c62dc0)
/usr/local/go/src/net/http/transport.go:1822 +0xec
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1238 +0x7cc

goroutine 318897 [runnable]:
internal/poll.runtime_pollWait(0x76f0e240, 0x72, 0x1208a000)
/usr/local/go/src/runtime/netpoll.go:173 +0x44
internal/poll.(*pollDesc).wait(0x12162744, 0x72, 0xffffff00, 0x83e9e8, 0xb20198)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:85 +0x8c
internal/poll.(*pollDesc).waitRead(0x12162744, 0x1208a000, 0x1000, 0x1000)
/usr/local/go/src/internal/poll/fd_poll_runtime.go:90 +0x2c
internal/poll.(*FD).Read(0x12162730, 0x1208a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/internal/poll/fd_unix.go:157 +0x148
net.(*netFD).Read(0x12162730, 0x1208a000, 0x1000, 0x1000, 0x651b88, 0x1641d8, 0x20394)
/usr/local/go/src/net/fd_unix.go:202 +0x38
net.(*conn).Read(0x11e8ca78, 0x1208a000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/local/go/src/net/net.go:176 +0x58
net/http.(*persistConn).Read(0x11c62e60, 0x1208a000, 0x1000, 0x1000, 0x11e52480, 0x2581a4, 0x40)
/usr/local/go/src/net/http/transport.go:1453 +0x174
bufio.(*Reader).fill(0x11fa2750)
/usr/local/go/src/bufio/bufio.go:100 +0x104
bufio.(*Reader).Peek(0x11fa2750, 0x1, 0x11ea4840, 0x122e5604, 0x7adae4, 0x0, 0xa1b40)
/usr/local/go/src/bufio/bufio.go:132 +0x28
net/http.(*persistConn).readLoop(0x11c62e60)
/usr/local/go/src/net/http/transport.go:1601 +0x14c
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1237 +0x7b0

goroutine 318917 [runnable]:
net/http.(*http2serverConn).runHandler(0x11c60480, 0x11e8d280, 0x12373480, 0x11f7a700)
/usr/local/go/src/net/http/h2_bundle.go:5739
created by net/http.(*http2serverConn).processHeaders
/usr/local/go/src/net/http/h2_bundle.go:5494 +0x3a4

goroutine 318898 [select]:
net/http.(*persistConn).writeLoop(0x11c62e60)
/usr/local/go/src/net/http/transport.go:1822 +0xec
created by net/http.(*Transport).dialConn
/usr/local/go/src/net/http/transport.go:1238 +0x7cc

trap 0x0
error 0x0
oldmask 0x0
r0 0xf0a7d0
r1 0x0
r2 0x0
r3 0x0
r4 0x0
r5 0x0
r6 0x11c1c03c
r7 0xf0
r8 0xf0a700
r9 0x17
r10 0xf0a500
fp 0x0
ip 0x0
sp 0x7ee1fd48
lr 0x392d4
pc 0x6bb00
cpsr 0x20000010
fault 0x0

@agnivade
Copy link
Contributor

Hi @dannin,

Please provide us with detailed information of how exactly you tested the application and a sample source code of the application so that we can debug and reproduce the issue.

Also, does the issue occur only in arm cpus ? What happens if you run it on amd64 ?

@agnivade agnivade added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 28, 2018
@dannin
Copy link
Author

dannin commented Oct 28, 2018

@agnivade
Unfortunately I can’t supply the application or source code.

Basically the service replies to http request with regular json. One of the handlers does a exec command.

We don’t have any mutexs in any of those code paths. As you can see from the stack trace, all of our go routines are in some wait / sleep.

Go routine 0 sticks out to me and 318913.

@agnivade
Copy link
Contributor

@dannin - Without a way to reproduce, it will be very hard to debug much. If you can find a way to extract sensitive parts of the application and still make it reproducible, that would be very helpful.

I will page @bradfitz to see if we can do anything here.

Also, have you seen this comment of mine ?

What happens if you run it on amd64 ?

It would be interesting to see if this happens only on arm. That would atleast eliminate some possibilities.

@agnivade agnivade changed the title Application hangs after a while net/http: application hangs after a while Oct 28, 2018
@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 28, 2018
@agnivade agnivade added this to the Unplanned milestone Oct 28, 2018
@dannin
Copy link
Author

dannin commented Oct 28, 2018

It has happened about 10-20 times so far in the past 6 months. We only run it on arm.

@dannin
Copy link
Author

dannin commented Oct 28, 2018

Actually to add, just realized, majority of the request are just proxied through.

proxy := SitemasterProxy{reverseProxy: httputil.NewSingleHostReverseProxy(proxyUrl)}
Then our handler just does:
proxy.ServeHTTP(w, r)

only one of the handlers does an exec.Command, which you can see in go routine 318913

@dannin
Copy link
Author

dannin commented Oct 30, 2018

Also we set GOMAXPROCS to 2, or board has one CPU.

@dannin
Copy link
Author

dannin commented Oct 30, 2018

@agnivade @bradfitz

@davecheney
Copy link
Contributor

Historically bug reports like these are related to poor error handling or data races. Are you able to run the various vet/check/lint programs over your code, they may help spot ineffective error handling. While it is not available on arm, if you can build and run your application on a platform supported by the race detector this may help find data race issues in your application.

@jared2501
Copy link

jared2501 commented Nov 2, 2018

Could this be related to #28204? I'm seeing hangs using the HTTP reverse proxy as well. I traced down that issue, but I think there's another (or that issue doesn't fully fix the problem).

@jared2501
Copy link

Hey @danin - can you try the patch I link in #28634 and see if it fixes your issue?

@agnivade
Copy link
Contributor

@dannin - We would need something more just the stack trace to debug this better. I understand that you don't run in on amd64. But does this repro too in amd64 ? I understand that there are no mutexes in the code paths, but could you run the code under race detector and see if there are any race issues ?

Also, could you run the code using GODEBUG=http2debug=1 and report the output when you see the hang ? That might be helpful.

@agnivade agnivade added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Dec 10, 2018
@jared2501
Copy link

Hey @agnivade - @prashantv managed to isolate a minimal repro of #28204 (see the ticket for the code). That issue could be the underlying cause for this issue since it uses the ReverseProxy.

@agnivade
Copy link
Contributor

Possibly. I will let @bradfitz weigh in.

@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Jan 10, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

5 participants