net/http: Server hangs and cannot receive any more requests when using ReverseProxy #20028
Comments
|
There might be a bug in Go, but it's not clear from this report that there is. This is a little high level and I don't have time at the moment to debug it. Have you discussed this yet on the mailing lists or other forums? Could you put your bug report in one self-contained file and put it on play.golang.org? (And remove anything unnecessary like graceful shutdown) I see some missing error checks and data races, but those are probably not relevant. Is the Windows cross-compiled part relevant? What are the goroutine stacks of the parent and child process when things get wedged? |
bradfitz
added
the
WaitingForInfo
bradfitz
changed the title
|
Also: any difference with Go 1.8.1? |
|
While I did mention that it was cross compiled, I did built it natively on windows (specifically windows 2012 server) and ran it. Same issue. I also did build it with 1.8.1 but they was with the original application that I was working on. I will return it again with this stripped down version. If I do it in 1 self contained source in play.golang.org, does it need to be able to execute? Just asking because it needs to spawn another process to receive the http reverseproxy as well as having an external tool to call the http endpoint. |
|
No, it doesn't need to be able to execute. That's just where we tend to share things. Please make the example as small as possible to still be able to demonstrate your problem. Remove anything unnecessary. |
|
Here is the self contained source https://play.golang.org/p/FCNCSSLGkD |
|
I compiled with go 1.8.1 on windows, and has the same issue |
|
Is the signal handling relevant? These commented-out lines definitely aren't relevant: Please delete everything that's not applicable to the bug report. |
|
I have removed the signal handler and irrelevant comments. |
|
Can you reproduce on Linux, or only on Windows? When it gets hung and you do a goroutine dump (SIGQUIT on Unix or runtime.Stack(buf, true) in SIGINT or http on Windows), what do you see the goroutines stuck doing? |
|
So far, I cannot reproduce in linux. How do i do a "goroutine dump"? I build the binary and run it. I don't have a debugger attached. I did use attach dlv before and looked through all threads and did a backtrace, but nothing seemed weird. |
|
See my comment. Wire up SIGINT to do something like: buf := make([]byte, 1<<20)
buf = buf[:runtime.Stack(buf, true)]
println(string(buf))On Linux you can just send SIGQUIT (Control-) to get a goroutine dump, but I don't think you can SIGQUIT on Windows, so you'll need to write that code above. |
|
PS C:\Shares\HenryShare\tmp> .\singlesource.exe --port=7100 goroutine 1 [chan receive]: goroutine 5 [syscall]: goroutine 7 [IO wait]: goroutine 8 [semacquire]: goroutine 9 [semacquire]: goroutine 10 [syscall, locked to thread]: goroutine 11 [syscall, locked to thread]: goroutine 16 [IO wait]: goroutine 33 [select]: 2017/04/19 13:28:49 [plugin stdout] /opt/emc/apps/golang/go/src/net/tcpsock.go:209 +0x50 |
|
Hello, is there an update? |
|
Any update would be posted here. There is no update, sorry. |
|
I think this might be #28168. |
agnivade
added
NeedsInvestigation
Please answer these questions before submitting your issue. Thanks!
What version of Go are you using (
go version)?$ go version
go version go1.7.5 linux/amd64
What operating system and processor architecture are you using (
go env)?$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/user/tmp/github/reverse-proxy-test"
GORACE=""
GOROOT="/opt/emc/apps/golang/go"
GOTOOLDIR="/opt/emc/apps/golang/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build252819066=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
What did you do?
Source: https://github.com/hleong25/reverse-proxy-test
I cross compiled from linux to windows:
GOOS=windows go build henryleong.com/...The binary will start an HTTP server and will act as a reverseproxy. I will refer to it as the agent. The agent will spawn the same binary but with a new port (agents port +1), read the stdout and stderr through separate pipes. The spawned binary will be referred to as plugin. The plugin will handle the actual HTTP request from the agent through a reverseproxy request.
On a separate machine, I do a loop to the agents endpoint:
while true; do curl http://<host>:<port>/greetings; done;What did you expect to see?
I expect the curl command will be processed very quickly.
What did you see instead?
Intermittently, after a short period, curl will hang waiting for the agents server to process the HTTP request.
I believe it has to do with the stdout/stderr pipes which is not being flushed and creates some sort of deadlock/race condition.
Workaround 1
If it gets into this situation, to get out of it, I would do a HTTP request to the plugin directly, and the agent starts processing the HTTP requests. Which after a while, the agent will get into this issue again and I would need to do a HTTP request to the plugin directly.
Workaround 2
If I modify the plugin to output something to stdout or to stderr periodically, the agents HTTP server will start responding to HTTP requests.
The text was updated successfully, but these errors were encountered: