net: A lookup performed by a canceled context might affect subsequent lookups #22724

tt · 2017-11-14T22:19:39Z

What version of Go are you using (`go version`)?

go version go1.9.2 darwin/amd64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/tt"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.9.2/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.9.2/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/v3/slr36r597rn0rrfzq_my6nv00000gn/T/go-build723069031=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

package main

import (
	"context"
	"fmt"
	"net"
	"testing"
)

func TestLookupIPAddr(t *testing.T) {
	ctx := context.Background()
	canceledCtx, cancel := context.WithCancel(ctx)
	cancel()

	var resolver net.Resolver

	_, err := resolver.LookupIPAddr(canceledCtx, "example.com")
	if fmt.Sprintf("%#v", err) != `&errors.errorString{s:"operation was canceled"}` {
		t.Fatalf("unexpected error: %q", err)
	}

	_, err = resolver.LookupIPAddr(ctx, "example.com")
	if err != nil {
		t.Fatalf("unexpected error: %q", err)
	}
}

(This is a reduced example; it happened to me through the use of net/http.)

What did you expect to see?

The test should produce no output.

Resolving the IP address using a canceled context should fail and the following attempt to resolve the IP address using a non-canceled context should succeed.

What did you see instead?

The test produces the following output:

--- FAIL: TestLookupIPAddr (0.00s)
	lookup_test.go:24: unexpected error: "lookup example.com on 89.233.43.71:53: dial udp 89.233.43.71:53: operation was canceled"
FAIL

That is, the second attempt to resolve the IP address using the non-canceled context returns the result of the previous invocation.

This is happening as parallel lookups for the same hosts only execute once:

go/src/net/lookup.go

Lines 220 to 224 in bb3be40

    
           // lookupGroup merges LookupIPAddr calls together for lookups 
        
           // for the same host. The lookupGroup key is is the LookupIPAddr.host 
        
           // argument. 
        
           // The return values are ([]IPAddr, error). 
        
           var lookupGroup singleflight.Group

... and following the first lookup, this goroutine exists:

1 @ 0x101298e 0x100535e 0x1188023 0x117c53d 0x118d9f6 0x1188982 0x118ae0c 0x115fade 0x105ca41
#	0x1188022	net.cgoLookupIP+0x72				/usr/local/Cellar/go/1.9.2/libexec/src/net/cgo_unix.go:212
#	0x117c53c	net.(*Resolver).lookupIP+0x12c			/usr/local/Cellar/go/1.9.2/libexec/src/net/lookup_unix.go:95
#	0x118d9f5	net.(*Resolver).(net.lookupIP)-fm+0x55		/usr/local/Cellar/go/1.9.2/libexec/src/net/lookup.go:187
#	0x1188981	net.glob..func10+0x51				/usr/local/Cellar/go/1.9.2/libexec/src/net/hook.go:19
#	0x118ae0b	net.(*Resolver).LookupIPAddr.func1+0x5b		/usr/local/Cellar/go/1.9.2/libexec/src/net/lookup.go:193
#	0x115fadd	internal/singleflight.(*Group).doCall+0x2d	/usr/local/Cellar/go/1.9.2/libexec/src/internal/singleflight/singleflight.go:93

Sleeping for just a millisecond between the two lookups allow the goroutine to complete and therefore resolves the IP address properly.

(It makes no difference if you re-initialize net.Resolver as the singleflight.Group variable is shared across instances.)

It's easy to fix this specific example (a sequential lookup) by "forgetting" the result if the context is canceled (similar to the solution applied in 77595e4 for #8602). I have a change for this and will submit it.

I do wonder if it would be more correct to never pass the context to the inner lookup; otherwise a goroutine that is canceled or times out will be able to affect other goroutines waiting for that same response.

The text was updated successfully, but these errors were encountered:

gopherbot · 2017-11-14T22:22:58Z

Change https://golang.org/cl/77670 mentions this issue: net: Forget lookups for canceled contexts

rsc · 2017-11-22T20:39:02Z

@bradfitz why did you reopen this? Does the CL not actually fix the problem?

bradfitz · 2017-11-22T20:49:48Z

It was reverted due to build breakages and code review comments.

gopherbot · 2017-11-23T12:58:17Z

Change https://golang.org/cl/79715 mentions this issue: net: Forget lookups for canceled contexts

odeke-em · 2017-12-06T10:24:13Z

Ping @tt, please take a look at the feedback that @mikioh posted plus the gentle ping from @bradfitz. Thank you.

Ragnaroek · 2018-01-22T10:12:55Z

This issue is very serious for us. We have a lot of canceled contexts and are flooded by this 'operation was canceled' error. For us this is a blocker. Can you please consider this to fix in Go 1.10?

ianlancetaylor · 2018-01-23T02:51:46Z

At the moment we don't even have an approved patch for this problem. The earlier attempt to fix it (https://golang.org/cl/77670) had to be rolled back, as described in the comments there. The problem exists in 1.9, so it's not new in 1.10. The simple fix for this--just checking whether the context has been canceled, as is done in https://golang.org/cl/79715 --will re-break #20703. We need a more sophisticated fix.

So I think it is too late to fix this for 1.10. Sorry. I will mark this as a release blocker for 1.11.

ianlancetaylor · 2018-03-15T18:10:04Z

This is a hard problem. As I just wrote on CL 79715, I see now that the earlier change (CL 45999) is not right. We should not be memoizing the result of a context cancellation. And simply reverting CL 45999 isn't enough. After all, another DNS lookup could arrive between the context cancellation and the lookupGroup.Forget, could get folded into the existing singleflight lookup, and could see the cancellation.

The problem is that singleflight is designed for tasks that either succeed or fail. It is not designed for tasks that get canceled, where the cancelation of one task should not affect another task using the same singleflight key. I think we need to separate the cancelation of the task from the cancelation of the singleflight operation.

gopherbot · 2018-03-15T18:56:33Z

Change https://golang.org/cl/100840 mentions this issue: net: don't let cancelation of a DNS lookup affect another lookup

ianlancetaylor · 2018-03-16T13:49:22Z

Reopening for 1.10.1

andybons · 2018-03-27T21:40:30Z

CL 100840 OK for Go 1.10.1

gopherbot · 2018-03-27T22:49:55Z

Change https://golang.org/cl/102787 mentions this issue: [release-branch.go1.10] net: don't let cancelation of a DNS lookup affect another lookup

gopherbot · 2018-03-28T19:49:20Z

Change https://golang.org/cl/103215 mentions this issue: [release-branch.go1.9] net: don't let cancelation of a DNS lookup affect another lookup

…fect another lookup Updates #8602 Updates #20703 Fixes #22724 Change-Id: I27b72311b2c66148c59977361bd3f5101e47b51d Reviewed-on: https://go-review.googlesource.com/100840 Run-TryBot: Ian Lance Taylor <iant@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Reviewed-on: https://go-review.googlesource.com/102787 Run-TryBot: Andrew Bonventre <andybons@golang.org> Reviewed-by: Ian Lance Taylor <iant@golang.org>

ghost · 2018-04-02T06:16:29Z

@andybons is this has been fixed in 1.10.1?

odeke-em · 2018-04-02T07:28:38Z

@andybons ama let you finish ;)

@AliGrab yes, please and Go1.10.1 was released last week https://twitter.com/golang/status/979377544196755456, please get it and try again.

devops-eatigo · 2018-06-12T10:33:27Z

@odeke-em Any one can confirm this fix is in the latest golang 1.10.3? how about 1.10.0?

Updated the golang version for golang/go#22724

gopherbot closed this as completed in 6a3d4be Nov 15, 2017

bradfitz added this to the Go1.10 milestone Nov 17, 2017

bradfitz reopened this Nov 17, 2017

bradfitz added the NeedsFix The path to resolution is known, but the work has not been done. label Nov 17, 2017

ianlancetaylor modified the milestones: Go1.10, Go1.11 Jan 3, 2018

ianlancetaylor added the release-blocker label Jan 23, 2018

bradfitz mentioned this issue Mar 2, 2018

net/http: All outgoing http requests are canceled if 1 dial times out #24194

Closed

gopherbot closed this as completed in bd85943 Mar 16, 2018

ianlancetaylor reopened this Mar 16, 2018

ianlancetaylor modified the milestones: Go1.11, Go1.10.1 Mar 16, 2018

andybons mentioned this issue Mar 27, 2018

all: Go 1.10.1 release status #24563

Closed

FiloSottile mentioned this issue Mar 27, 2018

all: Go 1.9.5 release status #24562

Closed

andybons mentioned this issue Mar 27, 2018

net: A lookup performed by a canceled context might affect subsequent lookups [1.9.5] #24569

Closed

andybons added CherryPickApproved Used during the release process for point releases and removed NeedsFix The path to resolution is known, but the work has not been done. labels Mar 27, 2018

ianlancetaylor mentioned this issue Mar 28, 2018

net: DialContext returns "dial tcp <addr>: context deadline exceeded" error #23648

Closed

andybons closed this as completed Mar 29, 2018

This was referenced Apr 3, 2018

operation was canceled during DNS lookup with remote_write prometheus/prometheus#3510

Closed

Use custom http.Client for AWS cortexproject/cortex#537

Merged

This was referenced Apr 10, 2018

cannot start flannel v0.10.0 flannel-io/flannel#977

Closed

Builds:Revert go version to 1.8.7 flannel-io/flannel#979

Closed

erdema-ft added a commit to Financial-Times/concept-rw-elasticsearch that referenced this issue Jun 15, 2018

Updated the golang version for golang/go#22724

c71bca0

geoah added a commit to Financial-Times/concept-rw-elasticsearch that referenced this issue Jun 20, 2018

Merge pull request #39 from Financial-Times/fix/dns-lookup-cancelled-ctx

51012fe

Updated the golang version for golang/go#22724

pingles mentioned this issue Feb 19, 2019

build with go 1.11 uswitch/kiam#218

Closed

golang locked and limited conversation to collaborators Jun 13, 2019

gopherbot added the FrozenDueToAge label Jun 13, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net: A lookup performed by a canceled context might affect subsequent lookups #22724

net: A lookup performed by a canceled context might affect subsequent lookups #22724

tt commented Nov 14, 2017

gopherbot commented Nov 14, 2017

rsc commented Nov 22, 2017

bradfitz commented Nov 22, 2017

gopherbot commented Nov 23, 2017

odeke-em commented Dec 6, 2017

Ragnaroek commented Jan 22, 2018 •

edited

ianlancetaylor commented Jan 23, 2018 •

edited

ianlancetaylor commented Mar 15, 2018

gopherbot commented Mar 15, 2018

ianlancetaylor commented Mar 16, 2018

andybons commented Mar 27, 2018

gopherbot commented Mar 27, 2018

gopherbot commented Mar 28, 2018

ghost commented Apr 2, 2018

odeke-em commented Apr 2, 2018

devops-eatigo commented Jun 12, 2018 •

edited

net: A lookup performed by a canceled context might affect subsequent lookups #22724

net: A lookup performed by a canceled context might affect subsequent lookups #22724

Comments

tt commented Nov 14, 2017

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

gopherbot commented Nov 14, 2017

rsc commented Nov 22, 2017

bradfitz commented Nov 22, 2017

gopherbot commented Nov 23, 2017

odeke-em commented Dec 6, 2017

Ragnaroek commented Jan 22, 2018 • edited

ianlancetaylor commented Jan 23, 2018 • edited

ianlancetaylor commented Mar 15, 2018

gopherbot commented Mar 15, 2018

ianlancetaylor commented Mar 16, 2018

andybons commented Mar 27, 2018

gopherbot commented Mar 27, 2018

gopherbot commented Mar 28, 2018

ghost commented Apr 2, 2018

odeke-em commented Apr 2, 2018

devops-eatigo commented Jun 12, 2018 • edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

Ragnaroek commented Jan 22, 2018 •

edited

ianlancetaylor commented Jan 23, 2018 •

edited

devops-eatigo commented Jun 12, 2018 •

edited