net/http: http.Transport leaks context from the request #50798

juliens · 2022-01-25T09:36:56Z

What version of Go are you using (`go version`)?

$ go version
go version go1.17 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE="on"
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/juliens/.cache/go-build"
GOENV="/home/juliens/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/juliens/dev/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/juliens/dev/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/juliens/.gvm/gos/go1.17"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/juliens/.gvm/gos/go1.17/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3799944390=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I use the http.Transport to make an HTTP request. I add a not empty struct in the request context.

You can reproduce it like this

What did you expect to see?

After the end of the request, the struct and the context are cleaned by the GC.

What did you see instead?

After the end of the request, the struct and the context are not cleaned by the GC.

Maybe this is the same issue as #43966

The text was updated successfully, but these errors were encountered:

gopherbot · 2022-01-25T09:43:31Z

Change https://golang.org/cl/380674 mentions this issue: net/http: fix memory leak in http.Transport

mknyszek · 2022-01-25T16:06:05Z

CC @neild

netrebel · 2022-01-29T07:35:24Z

We seem to be having the same problem. In long running instances, memory utilization is going up every day reaching 80-90% in ~12 days. We started seeing this issue when we upgraded from Go 1.11.3 (no modules) to 1.17 (with go-modules), before the Memory would stay under 30%. Servers have 4GBs.

When I run pprof I see something similar to the comment in the ticket referenced:

      flat  flat%   sum%        cum   cum%
    1.50MB  5.07%  5.07%    12.04MB 40.70%  net/http.(*Transport).dialConn
         0     0%  5.07%    12.04MB 40.70%  net/http.(*Transport).dialConnFor
    8.03MB 27.14% 32.21%     8.03MB 27.14%  bufio.NewWriterSize (inline)
    6.02MB 20.36% 52.57%     6.02MB 20.36%  bufio.NewReaderSize (inline)

luca147 · 2022-12-16T15:06:15Z

Same issue, there is some news about this?

netrebel · 2022-12-16T17:48:29Z

We seem to be having the same problem. In long running instances, memory utilization is going up every day reaching 80-90% in ~12 days. We started seeing this issue when we upgraded from Go 1.11.3 (no modules) to 1.17 (with go-modules), before the Memory would stay under 30%. Servers have 4GBs.

When I run pprof I see something similar to the comment in the ticket referenced:
      flat  flat%   sum%        cum   cum%
    1.50MB  5.07%  5.07%    12.04MB 40.70%  net/http.(*Transport).dialConn
         0     0%  5.07%    12.04MB 40.70%  net/http.(*Transport).dialConnFor
    8.03MB 27.14% 32.21%     8.03MB 27.14%  bufio.NewWriterSize (inline)
    6.02MB 20.36% 52.57%     6.02MB 20.36%  bufio.NewReaderSize (inline)

In case this helps someone, this other ticket was closed with a comment explaining how to avoid this issue: newrelic/go-agent#447 (comment)

luca147 · 2022-12-17T14:53:41Z

Well I spotted in my case the solution, I have a global scope context that are aware when a term signal come and all of my workers need to be stopped when it comes. So every time a job start I create a new context with cancel and defer it. With this simple change the leak over bufio.NewWriterSize/bufio.NewReaderSize into transport disappear.

urenner-n2i · 2023-05-04T12:35:39Z

Any update on this issue?
When connecting to new hosts very often, the memory consumption increases very quickly, and never decreases.

nikita21 · 2023-06-05T10:14:29Z

We are seeing similar issue when we upgraded from Go 1.13 (no modules) to 1.17 (with go-modules). In our worker instances, memory utilization is going up every day reaching 80-90% in ~8 days. Even tried upgrading to 1.19 but issue still persists.

xsteadfastx · 2023-06-19T15:38:05Z

my pprof looks exactly the same. are there any workarounds? what can i do?

…dTrip Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Fixes golang#50798

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Fixes golang#50798

gopherbot · 2023-07-22T11:37:11Z

Change https://go.dev/cl/512196 mentions this issue: net/http: clear reference to the request context after transport getConn

AlexanderYastrebov · 2023-07-22T20:56:57Z

Request context is used to dial connection and there is also #59017 (comment) that suggests that it should not be used for that.

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Updates golang#50798

AlexanderYastrebov · 2023-08-03T08:46:46Z

#61524 only fixes request context leakage but does not fix *persistConn leaks (seen as bufio.NewWriterSize and bufio.NewReaderSize on the pictures) which I plan to address separately.

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Updates #50798 Change-Id: I9a681ac0f222be56571fa768700220f6b5ee0888 GitHub-Last-Rev: fd6c83a GitHub-Pull-Request: #61524 Reviewed-on: https://go-review.googlesource.com/c/go/+/512196 Auto-Submit: Matthew Dempsky <mdempsky@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

Transport getConn creates wantConn w, tries to obtain idle connection for it based on the w.key and, when there is no idle connection, puts wantConn into idleConnWait wantConnQueue. Then getConn dials connection for w in a goroutine and blocks. After dial succeeds getConn unblocks and returns connection to the caller. At this point w is stored in the idleConnWait and will not be evicted until another wantConn with the same w.key is requested or alive connection returned into the idle pool which may not happen e.g. if server closes the connection. The problem is that even after tryDeliver succeeds w references persistConn wrapper that allocates bufio.Reader and bufio.Writer and prevents them from being garbage collected. To fix the problem this change removes persistConn and error references from wantConn and delivers them via channel to getConn. This way wantConn could be kept in wantConnQueues arbitrary long. Fixes golang#43966 Fixes golang#50798

gopherbot · 2023-08-23T00:49:36Z

Change https://go.dev/cl/522095 mentions this issue: net/http: remove persistConn reference from wantConn

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Updates golang#50798 Change-Id: I9a681ac0f222be56571fa768700220f6b5ee0888 GitHub-Last-Rev: fd6c83a GitHub-Pull-Request: golang#61524 Reviewed-on: https://go-review.googlesource.com/c/go/+/512196 Auto-Submit: Matthew Dempsky <mdempsky@google.com> Run-TryBot: Ian Lance Taylor <iant@golang.org> Reviewed-by: Matthew Dempsky <mdempsky@google.com> Reviewed-by: Damien Neil <dneil@google.com> TryBot-Result: Gopher Robot <gobot@golang.org>

Transport getConn creates wantConn w, tries to obtain idle connection for it based on the w.key and, when there is no idle connection, puts wantConn into idleConnWait wantConnQueue. Then getConn dials connection for w in a goroutine and blocks. After dial succeeds getConn unblocks and returns connection to the caller. At this point w is stored in the idleConnWait and will not be evicted until another wantConn with the same w.key is requested or alive connection returned into the idle pool which may not happen e.g. if server closes the connection. The problem is that even after tryDeliver succeeds w references persistConn wrapper that allocates bufio.Reader and bufio.Writer and prevents them from being garbage collected. To fix the problem this change removes persistConn and error references from wantConn and delivers them via channel to getConn. This way wantConn could be kept in wantConnQueues arbitrary long. Fixes golang#43966 Fixes golang#50798

juliens mentioned this issue Jan 25, 2022

net/http: fix memory leak in http.Transport #50799

Open

mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jan 25, 2022

mknyszek added this to the Backlog milestone Jan 25, 2022

tomMoulard mentioned this issue Jul 25, 2022

Add traffic size metrics traefik/traefik#9208

Merged

2 tasks

AlexanderYastrebov added a commit to AlexanderYastrebov/go that referenced this issue Jul 22, 2023

net/http: clear reference to the request context after transport getConn

e17d3af

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Fixes golang#50798

AlexanderYastrebov mentioned this issue Jul 22, 2023

net/http: clear reference to the request context after transport getConn #61524

Closed

AlexanderYastrebov mentioned this issue Jul 22, 2023

net/http: setting a timeout on http requests that use TLS can result in excessive requests to server #59017

Closed

AlexanderYastrebov added a commit to AlexanderYastrebov/go that referenced this issue Jul 25, 2023

net/http: clear reference to the request context after transport getConn

fd6c83a

Clears wannConn ctx and prevents pending dialConnFor after connection delivered or canceled. Updates golang#50798

AlexanderYastrebov mentioned this issue Aug 3, 2023

net/http: memory leak due to a large number of items in transport.reqCanceler #61708

Closed

AlexanderYastrebov mentioned this issue Aug 23, 2023

net/http: remove persistConn reference from wantConn #62227

Closed

gopherbot closed this as completed in 8590f0a Mar 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: http.Transport leaks context from the request #50798

net/http: http.Transport leaks context from the request #50798

juliens commented Jan 25, 2022 •

edited

gopherbot commented Jan 25, 2022

mknyszek commented Jan 25, 2022

netrebel commented Jan 29, 2022

luca147 commented Dec 16, 2022

netrebel commented Dec 16, 2022

luca147 commented Dec 17, 2022

urenner-n2i commented May 4, 2023

nikita21 commented Jun 5, 2023

xsteadfastx commented Jun 19, 2023

gopherbot commented Jul 22, 2023

AlexanderYastrebov commented Jul 22, 2023

AlexanderYastrebov commented Aug 3, 2023 •

edited

gopherbot commented Aug 23, 2023

net/http: http.Transport leaks context from the request #50798

net/http: http.Transport leaks context from the request #50798

Comments

juliens commented Jan 25, 2022 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot commented Jan 25, 2022

mknyszek commented Jan 25, 2022

netrebel commented Jan 29, 2022

luca147 commented Dec 16, 2022

netrebel commented Dec 16, 2022

luca147 commented Dec 17, 2022

urenner-n2i commented May 4, 2023

nikita21 commented Jun 5, 2023

xsteadfastx commented Jun 19, 2023

gopherbot commented Jul 22, 2023

AlexanderYastrebov commented Jul 22, 2023

AlexanderYastrebov commented Aug 3, 2023 • edited

gopherbot commented Aug 23, 2023

juliens commented Jan 25, 2022 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

AlexanderYastrebov commented Aug 3, 2023 •

edited