Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net/http: Transport leaks context.Contexts when limited by MaxConnsPerHost #33850

Closed
bcmills opened this issue Aug 26, 2019 · 2 comments
Closed

net/http: Transport leaks context.Contexts when limited by MaxConnsPerHost #33850

bcmills opened this issue Aug 26, 2019 · 2 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Milestone
Go1.13

Comments

@bcmills
Copy link
Contributor

bcmills commented Aug 26, 2019

The following test case fails at go1.13rc1 due to a leak introduced in CL 184262. The leak occurs due to persistConn pointers entering the connsPerHostWait queue and remaining reachable via that queue until a connection to the host is broken: if connections are never broken, the queue is never cleared.

type countedContext struct {
	context.Context
}

type contextCounter struct {
	atomicCount int64
}

func (cc *contextCounter) Track(ctx context.Context) context.Context {
	counted := new(countedContext)
	counted.Context = ctx
	atomic.AddInt64(&cc.atomicCount, 1)
	runtime.SetFinalizer(counted, cc.decrement)
	return counted
}

func (cc *contextCounter) decrement(*countedContext) {
	atomic.AddInt64(&cc.atomicCount, -1)
}

func (cc *contextCounter) Count() int64 {
	return atomic.LoadInt64(&cc.atomicCount)
}

func TestTransportPersistConnContextLeakWithMaxConnsPerHost(t *testing.T) {
	defer afterTest(t)

	ts := httptest.NewServer(HandlerFunc(func(w ResponseWriter, r *Request) {
		time.Sleep(time.Millisecond)
		w.WriteHeader(StatusOK)
	}))
	defer ts.Close()

	c := ts.Client()
	c.Transport.(*Transport).MaxConnsPerHost = 1

	running := make(chan bool, 2)
	defer func() {
		for n := cap(running); n > 0; n-- {
			running <- true
		}
	}()

	body := []byte("Hello")
	var cc contextCounter
	prevCount := cc.Count()
	for i := 0; ; i++ {
		n := cc.Count()
		if n < prevCount {
			break
		}
		if i >= 1<<12 {
			t.Fatalf("Count of allocated client context.Contexts (%d) did not decrease after %d Do / GC iterations.", n, i)
		}
		prevCount = n

		running <- true
		go func() {
			defer func() { <-running }()
			ctx := cc.Track(context.Background())
			req, err := NewRequest("POST", ts.URL, bytes.NewReader(body))
			if err != nil {
				t.Error(err)
			}
			_, err = c.Do(req.WithContext(ctx))
			if err != nil {
				t.Errorf("Do failed with error: %v", err)
			}
		}()

		if t.Failed() {
			return
		}
		runtime.GC()
	}
}

The test passes with that CL reverted, but since that CL fixed a significant deadlock (#32336), I intend to fix the leak (hopefully today) rather than reverting the CL.

CC @bradfitz
@bcmills bcmills added NeedsFix The path to resolution is known, but the work has not been done. release-blocker labels Aug 26, 2019
@bcmills bcmills added this to the Go1.13 milestone Aug 26, 2019
@bcmills bcmills self-assigned this Aug 26, 2019
@gopherbot
Copy link

Change https://golang.org/cl/191964 mentions this issue: net/http: fix wantConnQueue memory leaks in Transport

@gopherbot
Copy link

Change https://golang.org/cl/191967 mentions this issue: [release-branch.go1.13] net/http: fix wantConnQueue memory leaks in Transport

gopherbot pushed a commit that referenced this issue Aug 27, 2019
[release-branch.go1.13] net/http: fix wantConnQueue memory leaks in T…
44a66ac 
…ransport

I'm trying to keep the code changes minimal for backporting to Go 1.13,
so it is still possible for a handful of entries to leak,
but the leaks are now O(1) instead of O(N) in the steady state.

Longer-term, I think it would be a good idea to coalesce idleMu with
connsPerHostMu and clear entries out of both queues as soon as their
goroutines are done waiting.

Cherry-picked from CL 191964.

Updates #33849
Updates #33850
Fixes #33878

Change-Id: Ia66bc64671eb1014369f2d3a01debfc023b44281
Reviewed-on: https://go-review.googlesource.com/c/go/+/191964
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
(cherry picked from commit 94bf9a8)
Reviewed-on: https://go-review.googlesource.com/c/go/+/191967
tomocy pushed a commit to tomocy/go that referenced this issue Sep 1, 2019
@tomocy
net/http: fix wantConnQueue memory leaks in Transport
141139c 
I'm trying to keep the code changes minimal for backporting to Go 1.13,
so it is still possible for a handful of entries to leak,
but the leaks are now O(1) instead of O(N) in the steady state.

Longer-term, I think it would be a good idea to coalesce idleMu with
connsPerHostMu and clear entries out of both queues as soon as their
goroutines are done waiting.

Fixes golang#33849
Fixes golang#33850

Change-Id: Ia66bc64671eb1014369f2d3a01debfc023b44281
Reviewed-on: https://go-review.googlesource.com/c/go/+/191964
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
t4n6a1ka pushed a commit to t4n6a1ka/go that referenced this issue Sep 5, 2019
@t4n6a1ka
net/http: fix wantConnQueue memory leaks in Transport
f2b63ab 
I'm trying to keep the code changes minimal for backporting to Go 1.13,
so it is still possible for a handful of entries to leak,
but the leaks are now O(1) instead of O(N) in the steady state.

Longer-term, I think it would be a good idea to coalesce idleMu with
connsPerHostMu and clear entries out of both queues as soon as their
goroutines are done waiting.

Fixes golang#33849
Fixes golang#33850

Change-Id: Ia66bc64671eb1014369f2d3a01debfc023b44281
Reviewed-on: https://go-review.googlesource.com/c/go/+/191964
Run-TryBot: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>
@golang golang locked and limited conversation to collaborators Aug 26, 2020
@rsc rsc unassigned bcmills Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done. release-blocker
Projects
None yet
Milestone
Go1.13
Development

No branches or pull requests
2 participants
@bcmills @gopherbot