Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: fatal error: concurrent map iteration and map write transport.go #58494

Closed
johnbalvin opened this issue Feb 13, 2023 · 9 comments
Closed

net/http: fatal error: concurrent map iteration and map write transport.go #58494

johnbalvin opened this issue Feb 13, 2023 · 9 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@johnbalvin
Copy link
Contributor

What version of Go are you using (go version)?

$ 1.20

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

set GO111MODULE= set GOARCH=amd64 set GOBIN= set GOCACHE=C:\Users\user\AppData\Local\go-build set GOENV=C:\Users\user\AppData\Roaming\go\env set GOEXE=.exe set GOEXPERIMENT= set GOFLAGS= set GOHOSTARCH=amd64 set GOHOSTOS=windows set GOINSECURE= set GOMODCACHE=C:\Users\user\go\pkg\mod set GONOPROXY= set GONOSUMDB= set GOOS=windows set GOPATH=C:\Users\user\go set GOPRIVATE= set GOPROXY=https://proxy.golang.org,direct set GOROOT=C:\Program Files\Go set GOSUMDB=sum.golang.org set GOTMPDIR= set GOTOOLDIR=C:\Program Files\Go\pkg\tool\windows_amd64 set GOVCS= set GOVERSION=go1.20 set GCCGO=gccgo set GOAMD64=v1 set AR=ar set CC=gcc set CXX=g++ set CGO_ENABLED=0 set GOMOD=D:\upwork\andrew\lawplus\go.mod set GOWORK= set CGO_CFLAGS=-O2 -g set CGO_CPPFLAGS= set CGO_CXXFLAGS=-O2 -g set CGO_FFLAGS=-O2 -g set CGO_LDFLAGS=-O2 -g set PKG_CONFIG=pkg-config set GOGCCFLAGS=-m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=C:\Users\user\AppData\Local\Temp\go-build3669119433=/tmp/go-build -gno-record-gcc-switches Output
$ go env

What did you do?

Making multiple requests at the same time.
the transport information I'm using is:

transport := &http.Transport{
		MaxIdleConnsPerHost: 30,
		DisableKeepAlives:   true,
		TLSClientConfig: &tls.Config{
			Renegotiation:      tls.RenegotiateOnceAsClient,
			InsecureSkipVerify: true,
		},
	}

Not a reproducable error because the code is big and it happens on random times

What did you expect to see?

Code runnign well

What did you see instead?

fatal error: concurrent map iteration and map write

goroutine 941361 [running]:
net/http.Header.sortedKeyValues(0xc003d58780, 0x1b4bb20?)
        /usr/local/go/src/net/http/header.go:179 +0x12a
net/http.Header.writeSubset(0xc013a1a960?, {0x1b4bb20, 0xc00fa89f00}, 0x0?, 0x0)
        /usr/local/go/src/net/http/header.go:201 +0xd1
net/http.(*Request).write(0xc01180e300, {0x1b4bb20, 0xc00fa89f00}, 0x0, 0xc003b4a300, 0x0)
        /usr/local/go/src/net/http/request.go:658 +0x8d6
net/http.(*persistConn).writeLoop(0xc013706480)
        /usr/local/go/src/net/http/transport.go:2413 +0x171
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d


goroutine 776511 [select]:
net/http.(*Transport).getConn(0xc006a0b040, 0xc065c66a40, {{}, 0xc000512240, {0x170d77c, 0x5}, {0xc00d9a5940, 0x1f}, 0x0})
        /usr/local/go/src/net/http/transport.go:1382 +0x5c6
net/http.(*Transport).roundTrip(0xc006a0b040, 0xc0085b8100)
        /usr/local/go/src/net/http/transport.go:590 +0x79e
net/http.(*Transport).RoundTrip(0xc0085b8100?, 0x1b51260?)
        /usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc0085b8000, {0x1b51260, 0xc006a0b040}, {0x8?, 0x1676500?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc0035c8c00, 0xc0085b8000, {0x0?, 0xf?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc0035c8c00, 0xc0085b8000)
        /usr/local/go/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc0085b8000, 0x1, 0x0, 0x4a817c800)
        /workspace/shared/requests/request.go:120 +0x665
lawplus/shared/requests.Get(0xc0085b8000, 0x1, 0x170d77c?, 0x39?)
        /workspace/shared/requests/request.go:27 +0xe6

goroutine 941344 [IO wait]:
internal/poll.runtime_pollWait(0x7f20965b6f28, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc003b2a500?, 0xc0045b1000?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc003b2a500, {0xc0045b1000, 0x1000, 0x1000})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc003b2a500, {0xc0045b1000?, 0xc005fc3700?, 0x61?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc012799490, {0xc0045b1000?, 0xc005fc3820?, 0x40d8e8?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc01234d170, {0xc0045b1000?, 0xc01234d170?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc015345090, {0x1b4da20, 0xc01234d170})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc015344e00, {0x1b51180?, 0xc012799490}, 0x0?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc015344e00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc015344e00, {0xc0045b6000, 0x1000, 0xc005fc3cc8?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc010f16900, {0xc0045b6000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc008f1ff80)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc008f1ff80, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc010f16900)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea

goroutine 941288 [select]:
net/http.(*persistConn).writeLoop(0xc013b13680)
        /usr/local/go/src/net/http/transport.go:2410 +0xf2
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d

goroutine 941298 [select]:
net/http.(*persistConn).writeLoop(0xc013b137a0)
        /usr/local/go/src/net/http/transport.go:2410 +0xf2
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1766 +0x173d

goroutine 940845 [IO wait]:
internal/poll.runtime_pollWait(0x7f20965b67a8, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc006a57300?, 0xc006b49800?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc006a57300, {0xc006b49800, 0x1800, 0x1800})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc006a57300, {0xc006b49800?, 0xc004278700?, 0x61?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc006851908, {0xc006b49800?, 0xc004278820?, 0x668120?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc01148b428, {0xc006b49800?, 0xc01148b428?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc006f0b090, {0x1b4da20, 0xc01148b428})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc006f0ae00, {0x1b51180?, 0xc006851908}, 0x0?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc006f0ae00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc006f0ae00, {0xc002ddc000, 0x1000, 0x1400600?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc013158120, {0xc002ddc000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc0079c98c0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc0079c98c0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc013158120)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea

goroutine 776510 [select]:
net/http.(*persistConn).roundTrip(0xc013b13680, 0xc0561e8000)
        /usr/local/go/src/net/http/transport.go:2638 +0x994
net/http.(*Transport).roundTrip(0xc00606e780, 0xc025ca2e00)
        /usr/local/go/src/net/http/transport.go:603 +0x7fa
net/http.(*Transport).RoundTrip(0xc025ca2e00?, 0x1b51260?)
        /usr/local/go/src/net/http/roundtrip.go:17 +0x19
net/http.send(0xc025ca2d00, {0x1b51260, 0xc00606e780}, {0x8?, 0x1676500?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:252 +0x5f7
net/http.(*Client).send(0xc004204fc0, 0xc025ca2d00, {0x4cb77f?, 0x9f876c?, 0x24b21c0?})
        /usr/local/go/src/net/http/client.go:176 +0x9b
net/http.(*Client).do(0xc004204fc0, 0xc025ca2d00)
        /usr/local/go/src/net/http/client.go:716 +0x8fb
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc025ca2d00, 0x0, 0x0, 0xdf8475800)
        /workspace/shared/requests/request.go:120 +0x665

goroutine 776506 [runnable]:
net/http.Header.Clone(...)
        /usr/local/go/src/net/http/header.go:106
net/http.cloneOrMakeHeader(0xc003d58780?)
        /usr/local/go/src/net/http/clone.go:69 +0x14c
net/http.(*Client).makeHeadersCopier(0xc003b4a360, 0xc01180e200)
        /usr/local/go/src/net/http/client.go:745 +0x45
net/http.(*Client).do(0xc003b4a360, 0xc01180e200)
        /usr/local/go/src/net/http/client.go:603 +0x276
net/http.(*Client).Do(...)
        /usr/local/go/src/net/http/client.go:582
lawplus/shared/requests.make(0xc01180e200, 0xb, 0xc006986090, 0x5d21dba00)
        /workspace/shared/requests/request.go:120 +0x665


goroutine 941297 [IO wait]:
internal/poll.runtime_pollWait(0x7f2096e33298, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc0035e7b00?, 0xc002ddac00?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc0035e7b00, {0xc002ddac00, 0x1300, 0x1300})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc0035e7b00, {0xc002ddac00?, 0xc002ddac05?, 0x11a?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc0119dd360, {0xc002ddac00?, 0x1b5b020?, 0xc013c85330?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc0077802e8, {0xc002ddac00?, 0xc0077802e8?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc013c85410, {0x1b4da20, 0xc0077802e8})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc013c85180, {0x1b51180?, 0xc0119dd360}, 0x1300?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc013c85180, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).Read(0xc013c85180, {0xc00424d000, 0x1000, 0x655f2f?})
        /usr/local/go/src/crypto/tls/conn.go:1288 +0x16f
net/http.(*persistConn).Read(0xc013b137a0, {0xc00424d000?, 0x405c66?, 0x60?})
        /usr/local/go/src/net/http/transport.go:1943 +0x4e
bufio.(*Reader).fill(0xc004d25ec0)
        /usr/local/go/src/bufio/bufio.go:106 +0xff
bufio.(*Reader).Peek(0xc004d25ec0, 0x1)
        /usr/local/go/src/bufio/bufio.go:144 +0x5d
net/http.(*persistConn).readLoop(0xc013b137a0)
        /usr/local/go/src/net/http/transport.go:2107 +0x1ac
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:1765 +0x16ea
goroutine 941142 [chan receive]:
net/http.(*persistConn).addTLS(0xc0137bd7a0, {0x1b59f68?, 0xc0035c8c90}, {0xc00d9a5940, 0x1b}, 0x0)
        /usr/local/go/src/net/http/transport.go:1550 +0x365
net/http.(*Transport).dialConn(0xc006a0b040, {0x1b59f68, 0xc0035c8c90}, {{}, 0xc000512240, {0x170d77c, 0x5}, {0xc00d9a5940, 0x1f}, 0x0})
        /usr/local/go/src/net/http/transport.go:1746 +0x11b7
net/http.(*Transport).dialConnFor(0x16f6773?, 0xc0081036b0)
        /usr/local/go/src/net/http/transport.go:1456 +0xb0
created by net/http.(*Transport).queueForDial
        /usr/local/go/src/net/http/transport.go:1425 +0x3ea


goroutine 941153 [IO wait]:
internal/poll.runtime_pollWait(0x7f2096e32c08, 0x72)
        /usr/local/go/src/runtime/netpoll.go:306 +0x89
internal/poll.(*pollDesc).wait(0xc004bdb500?, 0xc002de5d40?, 0x0)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:84 +0x32
internal/poll.(*pollDesc).waitRead(...)
        /usr/local/go/src/internal/poll/fd_poll_runtime.go:89
internal/poll.(*FD).Read(0xc004bdb500, {0xc002de5d40, 0x240, 0x240})
        /usr/local/go/src/internal/poll/fd_unix.go:167 +0x299
net.(*netFD).Read(0xc004bdb500, {0xc002de5d40?, 0x0?, 0x205?})
        /usr/local/go/src/net/fd_posix.go:55 +0x29
net.(*conn).Read(0xc00ef38df0, {0xc002de5d40?, 0x47c725?, 0x0?})
        /usr/local/go/src/net/net.go:183 +0x45
crypto/tls.(*atLeastReader).Read(0xc02127ab40, {0xc002de5d40?, 0xc02127ab40?, 0x0?})
        /usr/local/go/src/crypto/tls/conn.go:788 +0x3d
bytes.(*Buffer).ReadFrom(0xc002801090, {0x1b4da20, 0xc02127ab40})
        /usr/local/go/src/bytes/buffer.go:202 +0x98
crypto/tls.(*Conn).readFromUntil(0xc002800e00, {0x1b51180?, 0xc00ef38df0}, 0xc004481710?)
        /usr/local/go/src/crypto/tls/conn.go:810 +0xe5
crypto/tls.(*Conn).readRecordOrCCS(0xc002800e00, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:617 +0x116
crypto/tls.(*Conn).readRecord(...)
        /usr/local/go/src/crypto/tls/conn.go:583
crypto/tls.(*Conn).readHandshake(0xc002800e00)
        /usr/local/go/src/crypto/tls/conn.go:1020 +0x6d
crypto/tls.(*Conn).clientHandshake(0xc002800e00, {0x1b59ef8, 0xc01414d860})
        /usr/local/go/src/crypto/tls/handshake_client.go:184 +0x257
crypto/tls.(*Conn).handshakeContext(0xc002800e00, {0x1b59f68, 0xc0035c8c90})
        /usr/local/go/src/crypto/tls/conn.go:1463 +0x32e
crypto/tls.(*Conn).HandshakeContext(...)
        /usr/local/go/src/crypto/tls/conn.go:1406
net/http.(*persistConn).addTLS.func2()
        /usr/local/go/src/net/http/transport.go:1544 +0x70
created by net/http.(*persistConn).addTLS
        /usr/local/go/src/net/http/transport.go:1540 +0x345

goroutine 941154 [select]:
crypto/tls.(*Conn).handshakeContext.func2()
        /usr/local/go/src/crypto/tls/conn.go:1439 +0x98
created by crypto/tls.(*Conn).handshakeContext
        /usr/local/go/src/crypto/tls/conn.go:1438 +0x205

When looking at the code where the error is being produced, I get the error is here:
image
so probally it's because go is trying to write/iterate to the header which is underline type is a map, and it's not using any mutex

image

image
@seankhliao
Copy link
Member

that's just the iteration, no map write.
are you sure http.Request isn't shared from within your code?

@seankhliao seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Feb 13, 2023
@seankhliao seankhliao changed the title affected/package: net/http fatal error: concurrent map iteration and map write transport.go net/http fatal error: concurrent map iteration and map write transport.go Feb 13, 2023
@seankhliao seankhliao changed the title net/http fatal error: concurrent map iteration and map write transport.go net/http: fatal error: concurrent map iteration and map write transport.go Feb 13, 2023
@johnbalvin
Copy link
Contributor Author

that's just the iteration, no map write. are you sure http.Request isn't shared from within your code?

I'm sharing the client, but the code where I share the client is synchronous, it shouldn't affect it

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Feb 13, 2023
@bcmills
Copy link
Contributor

bcmills commented Feb 13, 2023

This sort of racy map operation is one of the kinds of races that the Go race detector diagnoses. If you run your program under that, it should print a stack trace for whatever goroutine is writing to the map.

Please do that, and let us know what the race detector reports.

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Feb 13, 2023
@johnbalvin
Copy link
Contributor Author

This sort of racy map operation is one of the kinds of races that the Go race detector diagnoses. If you run your program under that, it should print a stack trace for whatever goroutine is writing to the map.

Please do that, and let us know what the race detector reports.

I'll do it, thank you

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Feb 13, 2023
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@tripang1702
Copy link

I have a same issue too, when I am trying to benchmarking with 400 virtual user / second for 40 second

@panjf2000
Copy link
Member

Could you provide a full example for us to reproduce this issue?

@tripang1702
Copy link

Could you provide a full example for us to reproduce this issue?

actually, I just found the solution of this issue for my case, that was happen because I have implement some defer function in my code. I don't know for detail, but that issue fixed after I remove that defer function.

@johnbalvin
Copy link
Contributor Author

Could you provide a full example for us to reproduce this issue?

For more context for my case, I was making multiple requests at the same time to a single domain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@bcmills @panjf2000 @gopherbot @seankhliao @johnbalvin @tripang1702