Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/http: don't read Request headers after ServeHTTP starts #14940

Closed
bronze1man opened this issue Mar 24, 2016 · 8 comments
Closed

net/http: don't read Request headers after ServeHTTP starts #14940

bronze1man opened this issue Mar 24, 2016 · 8 comments
Milestone

Comments

@bronze1man
Copy link
Contributor

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?
    go version go1.6 linux/amd64
  2. What operating system and processor architecture are you using (go env)?
    GOARCH="amd64"
    GOBIN=""
    GOEXE=""
    GOHOSTARCH="amd64"
    GOHOSTOS="linux"
    GOOS="linux"
    GOPATH=""
    GORACE=""
    GOROOT="/usr/local/go"
    GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
    GO15VENDOREXPERIMENT="1"
    CC="gcc"
    GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0"
    CXX="g++"
    CGO_ENABLED="1"

This issues is happend on a product api server with qps about 100/s.
It will crash the process.

fatal error: concurrent map read and map write

goroutine 226231 [running]:
runtime.throw(0x1444b00, 0x21)
    /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc820fed438 sp=0xc820fed420
runtime.mapaccess1_faststr(0x12550e0, 0xc820858bd0, 0x12e2cd0, 0xa, 0x2daa800)
    /usr/local/go/src/runtime/hashmap_fast.go:202 +0x5b fp=0xc820fed498 sp=0xc820fed438
net/http.(*Request).wantsClose(0xc8207fc000, 0xc820859000)
    /usr/local/go/src/net/http/request.go:1071 +0x6d fp=0xc820fed510 sp=0xc820fed498
net/http.(*chunkWriter).writeHeader(0xc8200fe440, 0x0, 0x0, 0x0)
    /usr/local/go/src/net/http/server.go:943 +0x7d1 fp=0xc820fed930 sp=0xc820fed510
net/http.(*chunkWriter).close(0xc8200fe440)
    /usr/local/go/src/net/http/server.go:286 +0x5d fp=0xc820fedba0 sp=0xc820fed930
net/http.(*response).finishRequest(0xc8200fe410)
    /usr/local/go/src/net/http/server.go:1252 +0x96 fp=0xc820fedbd0 sp=0xc820fedba0
net/http.(*conn).serve(0xc822e27680)
    /usr/local/go/src/net/http/server.go:1476 +0xf68 fp=0xc820fedf98 sp=0xc820fedbd0
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820fedfa0 sp=0xc820fedf98
created by net/http.(*Server).Serve
    /usr/local/go/src/net/http/server.go:2137 +0x44e

...

It generated a 2MB panic log,but

  • i can not found hashmap_fast.go in the rest of the log.
  • All code run that is write by me in the panic log do not read/write http.Request.Header in the log.
  • It happens once per day.

I may do something bad. but how can i found the bad thing?
What is the possible bad thing ?

@randall77
Copy link
Contributor

Can you run your server with the race detector (-race) enabled? It may give us a better idea of where this race is coming from.

@bronze1man
Copy link
Contributor Author

Yes,I can.
It may take time to add -race into our DevOps system.

@bradfitz
Copy link
Contributor

Is your http.Handler code modifying the (*http.Request).Header map?

@bradfitz bradfitz changed the title [net/http] fatal error: concurrent map read and map write crash in net/http package. net/http: concurrent map read and map write crash Mar 24, 2016
@bronze1man
Copy link
Contributor Author

Is ok to write (*http.Request).Header map in the http.handler callback and in the http.Handler thread?
I just found some handlers in reverse proxy part can write that header.

@bradfitz
Copy link
Contributor

I suppose net/http shouldn't read from Request.Header after ServeHTTP has been started.

@bronze1man
Copy link
Contributor Author

Thanks for help.
I think I should apologize that I waste your time for understanding what happened.

I think I may found the reason of this crash.
I find some code in reverse proxy part that write (*http.Request).Header map in another thread which may did it after the http.handler callback return.

I will try to fix it and wait for crash for a week, and keep this issues open.

@bradfitz
Copy link
Contributor

Not a waste of time. We could change net/http to look at what it needs to in the request headers before starting the user's ServeHTTP func and be more robust against this in general.

@bradfitz bradfitz self-assigned this Mar 25, 2016
@bradfitz bradfitz added this to the Go1.7 milestone Mar 25, 2016
@bradfitz bradfitz changed the title net/http: concurrent map read and map write crash net/http: don't read Request headers after ServeHTTP starts Mar 25, 2016
@gopherbot
Copy link

CL https://golang.org/cl/21530 mentions this issue.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants