Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http/httputil: proxied streaming responses appear to lose some data on the way to the client #51646

Closed
rolandjitsu opened this issue Mar 14, 2022 · 10 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@rolandjitsu
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.16.4 darwin/arm64

Does this issue reproduce with the latest release?

I'm about to test that today.

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/rolandgroza/Library/Caches/go-build"
GOENV="/Users/rolandgroza/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/rolandgroza/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/rolandgroza/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/Cellar/go/1.16.4/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/Cellar/go/1.16.4/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.16.4"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/rolandgroza/Work/echo-streaming-bug/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/_9/htkb4h950nv8qctq6pfxc6mr0000gn/T/go-build365273782=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I've setup a simple echo server and I'm using streaming responses to stream data to clients through another proxy server.

Everything seemed to work fine, but I later noticed that after some time (25 - 30 mins), browser clients would get a:

SyntaxError: Unexpected token { in JSON at position 119
    at streamData (<anonymous>:20:40)

I initially thought the issue was in labstack/echo#2125. But then I also tried to use the net/http/httputil reverse proxy directly and I could still reproduce the error. So I think the issue could either be in the reverse proxy, the net/http server or could be in the browser.

I've tried my best to describe all the steps to reproduce the error at transcelestial/echo-streaming-bug.

I'll try to remove echo and use a simple server to check if the issue still persists.

What did you expect to see?

I don't expect to see any errors or data loss on the client end.

What did you see instead?

I get

SyntaxError: Unexpected token { in JSON at position 119
    at streamData (<anonymous>:20:40)

after ~ 25 mins.

@mengzhuo
Copy link
Contributor

I don't see this is a Go project's issue.
Could you provide a more concise example in https://go.dev/play?

@rolandjitsu
Copy link
Author

I don't see this is a Go project's issue. Could you provide a more concise example in https://go.dev/play?

Hi. Well, the issue only happens in browser clients. So data sent by the Go server is somehow corrupted along the way. I wouldn't be able to reproduce in a play env.

I'm not certain if this is a browser bug or something in the reverse proxy or net lib. But I can reproduce in different browsers, which led me to believe that the issue must be on the server side.

Is there something wrong with the details I've shared in that repo? Are you not able to reproduce?

@mengzhuo
Copy link
Contributor

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions


https://github.com/transcelestial/echo-streaming-bug/blob/74b734755472389d5dac7d00aab1cc4da75f3c9f/cmd/client/main.go#L57-L61

	for {
		line, err := reader.ReadBytes('\n')
		if err != nil {
			log.Fatal(err)
		}

It appears that your client can't split messages correctly when your data/content contains \n.

@rolandjitsu
Copy link
Author

rolandjitsu commented Mar 14, 2022

@mengzhuo there may be an actual bug within one of the net libs. I would assume someone would help look into it. I'm not asking a question as to how to stream responses or read them as that's already done and working.

I'm experiencing an actual issue where the browser based clients do not receive the full data. I can look into it myself, I just need some guidance so that I know what to look for or where I should be looking.

I've already made some effort in trying to explain as best as I could how to reproduce. I would at least expect the same effort on your end to check it out and tell me with certainty that the issue is probably on the client side. I'm definitely aware this is an open source project and you or other contributors have no obligation to help, but if this is a bug, I'm sure you'd like to fix it at some point.

I would appreciate it if you didn't close this issue as I think it's relevant. For context, it's very similar to #27816.


The referenced client actually works just fine. I was unable to replicate the issue with it. So the issue seems to happen strictly when using a browser. So I wanted to verify that there's no issue on the server side and then report the bug in the appropriate channels for each browser where it happens.

@mengzhuo
Copy link
Contributor

Sure, This triage is based on my understanding of your code.
I will reopen it if you think it's a bug.

cc @neild net/http owner

@mengzhuo mengzhuo reopened this Mar 14, 2022
@rolandjitsu
Copy link
Author

Sure, This triage is based on my understanding of your code. I will reopen it if you think it's a bug.

cc @neild net/http owner

I appreciate that! Thank you! As mentioned, I'm more than happy to look into it if someone points me in the right direction.

@mengzhuo mengzhuo added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 14, 2022
@iquanxin
Copy link

我也遇到类似的情况。使用 rust 反向代理测试正常,但是使用 httputil 就会出错〜,模拟发送http请求也不行。像是go的 http 实现有问题或有默认的参数设置问题,目前找不到原因。

@rolandjitsu
Copy link
Author

I've reported the issue at https://bugs.chromium.org/p/chromium/issues/detail?id=1305928 as well. I've also ran more tests just using net/http and net/http/httputil to see if the issue could be in echo, but I could still get failures.

So that leaves either the browsers or net/http (since I can get failures from net/http w/o using a proxy).

@seankhliao
Copy link
Member

Given that this doesn't need a reverse proxy to reproduce, it doesn't look like an issue with net/http/httputil
Adding some extra logging, Go sends the output within a single write.
On the browser/js side, it's a data stream, and a single json message can be spread over 2 reads, eg I got:

{"pong":true,"reason":"the bug does not seem to be here :( where is it then? who knows? at this point, i have no clue! i'm lost 

and

now. i have no clue what to do. i need help. i don't know what i'm doing. where should i look for help? where could the bug be? is there even a bug? right now, i'm questioning everything!","date":"2022-03-14T15:20:09Z","quote":"dumbass! (that 70's show)"}

spread over 2 reads

Closing as this doesn't appear to be a bug with Go (or Chrome for that matter, just improper use of streaming APIs)

@rolandjitsu
Copy link
Author

@seankhliao ok, got it. I was under the impression that the browser would read line by line at every occurrence of a \n char.

But I guess you're right since I found a handling text line by line example which illustrates how to handle chunks not ending on line boundaries. It was just not very clear from the docs that the readable stream doesn't necessarily split chunks at line endings.

Thanks for taking the time to check the code I shared and giving more insight into the issue!

@golang golang locked and limited conversation to collaborators Mar 15, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants