net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697

islinwb · 2023-01-09T09:25:58Z

What version of Go are you using (`go version`)?

$ go version
go version go1.18.7 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/root/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/root/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.18.7"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/root/gop/test/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2745863489=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I use func (c *Client) Do(req *Request) to make a http GET request with the pure Go dns resolver.

/etc/resolv.conf:

# Generated by NetworkManager
search openstacklocal novalocal
nameserver A.A.A.A 
nameserver B.B.B.B
nameserver C.C.C.C

It queries A record successfully from nameserver A.A.A.A in the first try.
However, it queries AAAA record form 3 nameserver with 2 attemps, ending in failure until timeout.

I check the source code and find it will always query A and AAAA records.

// LookupIPAddr looks up host using the local resolver.
// It returns a slice of that host's IPv4 and IPv6 addresses.
func (r *Resolver) LookupIPAddr(ctx context.Context, host string) ([]IPAddr, error) {
	return r.lookupIPAddr(ctx, "ip", host)
}

func (r *Resolver) goLookupIPCNAMEOrder(ctx context.Context, network, name string, order hostLookupOrder) (addrs []IPAddr, cname dnsmessage.Name, err error) {
//...
	qtypes := []dnsmessage.Type{dnsmessage.TypeA, dnsmessage.TypeAAAA}
	switch ipVersion(network) {
	case '4':
		qtypes = []dnsmessage.Type{dnsmessage.TypeA}
	case '6':
		qtypes = []dnsmessage.Type{dnsmessage.TypeAAAA}
	}
// ...
}

Is this expected?

What did you expect to see?

it returns after getting A record successfully. (or is there a way to configure this)

What did you see instead?

it tries to query AAAA record until timeout

The text was updated successfully, but these errors were encountered:

bcmills · 2023-01-09T15:48:37Z

Does the call to Do itself succeed?

(Is it just the AAAA DNS request that is blocked, or the entire call to Do?)

(CC @neild)

islinwb · 2023-01-10T09:17:28Z

@bcmills the Do blocked. It tries to query AAAA record until timeout.
more detail on my case:

nameserver A.A.A.A returns A record successfully and refused for AAAA record query
nameserver B.B.B.B and C.C.C.C are not accessible.
no AAAA record for the domain.

It takes 20+ seconds to finish Do (2 attempt * 2 nameserver * 5 second).
In the end, Do returns expected Get result but it takes too much time.

bcmills · 2023-01-10T13:44:29Z

In the current implementation, Dialer.DialContext first resolves all addresses for the given host, then partitions them by IPv4 / IPv6, then dials those in parallel per the configured FallbackDelay:
https://cs.opensource.google/go/go/+/master:src/net/dial.go;l=422-438;drc=8232a09e3ed7d315a90ac059ee542ecaf0f6b4c2

In order to use the A record without waiting for the AAAA records to resolve, we would have to instead dial each address as it is returned. That would require a very different internal concurrency model from what is implemented today.

jfesler · 2023-01-10T14:58:59Z

That would also push a lot of traffic towards IPv4 (possibly NAT'd) instead of using IPv6.

Any DNS server giving zero response to the AAAA lookup is broken. Fix it.

The more sophisticated models of Happy Eyeballs will wait at least a /little/ on the slower response; not necessarily giving up but trying to still ensure the currently favored protocol is used first.

Happy Eyeballs: Success with Dual-Stack Hosts [rfc6555, 2012],
Happy Eyeballs Version 2: Better Connectivity Using Concurrency [rfc8305 2017]

leosocy · 2023-05-18T07:49:22Z

In the current implementation, Dialer.DialContext first resolves all addresses for the given host, then partitions them by IPv4 / IPv6, then dials those in parallel per the configured FallbackDelay: https://cs.opensource.google/go/go/+/master:src/net/dial.go;l=422-438;drc=8232a09e3ed7d315a90ac059ee542ecaf0f6b4c2

In order to use the A record without waiting for the AAAA records to resolve, we would have to instead dial each address as it is returned. That would require a very different internal concurrency model from what is implemented today.

But according to the implementation of the partition method, the primaries after partition depend on whether the first record of the addrs slice is IPv4 or IPv6, If the first is IPv4 then the primaries are IPv4 slice, otherwise primaries are IPv6 slice. I don't know if this is as expected.

mateusz834 · 2023-12-19T13:23:47Z

From #64783 this seem to happen while using non-recursive DNS resolvers in resolv.conf

gopherbot · 2024-01-12T07:40:18Z

Change https://go.dev/cl/550435 mentions this issue: net: Prevent unintended retries upon receiving an empty answer response from the DNS server.

gopherbot · 2024-02-20T03:19:39Z

Change https://go.dev/cl/565295 mentions this issue: net: prevent unintended retries upon receiving an empty answer response from the DNS server.

gopherbot · 2024-02-20T03:22:52Z

Change https://go.dev/cl/565296 mentions this issue: net: prevent unintended retries upon receiving an empty answer response from the DNS server.

kkHAIKE · 2024-03-01T03:13:30Z

@gopherbot please consider this for backport to 1.22

gopherbot · 2024-03-01T03:14:13Z

Backport issue(s) opened: #66050 (for 1.22).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://go.dev/wiki/MinorReleases.

islinwb changed the title ~~net.http Do() query DNS A recrod succeeds but try to query AAAA record until timeout~~ net/http Do() query DNS A recrod succeeds but try to query AAAA record until timeout Jan 9, 2023

bcmills added this to the Backlog milestone Jan 9, 2023

bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 9, 2023

bcmills changed the title ~~net/http Do() query DNS A recrod succeeds but try to query AAAA record until timeout~~ net/http: Client.Do queries DNS AAAA record until timeout even though A record succeeds Jan 9, 2023

bcmills added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jan 10, 2023

bcmills changed the title ~~net/http: Client.Do queries DNS AAAA record until timeout even though A record succeeds~~ net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds Jan 10, 2023

seankhliao mentioned this issue Dec 18, 2023

net: Go's DNS resolver retry AAAA request to next nameserver even A response has records #64783

Closed

gopherbot closed this as completed in aaf8e84 Feb 19, 2024

dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Mar 1, 2024

dmitshur modified the milestones: Backlog, Go1.23 Mar 1, 2024

gopherbot mentioned this issue Mar 1, 2024

net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds [1.22 backport] #66050

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697

net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697

islinwb commented Jan 9, 2023 •

edited

bcmills commented Jan 9, 2023

islinwb commented Jan 10, 2023 •

edited

bcmills commented Jan 10, 2023 •

edited

jfesler commented Jan 10, 2023

leosocy commented May 18, 2023

mateusz834 commented Dec 19, 2023

gopherbot commented Jan 12, 2024

gopherbot commented Feb 20, 2024

gopherbot commented Feb 20, 2024

kkHAIKE commented Mar 1, 2024

gopherbot commented Mar 1, 2024

net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697

net/http: Client.Do blocks on DNS AAAA record until timeout even after A record succeeds #57697

Comments

islinwb commented Jan 9, 2023 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

bcmills commented Jan 9, 2023

islinwb commented Jan 10, 2023 • edited

bcmills commented Jan 10, 2023 • edited

jfesler commented Jan 10, 2023

leosocy commented May 18, 2023

mateusz834 commented Dec 19, 2023

gopherbot commented Jan 12, 2024

gopherbot commented Feb 20, 2024

gopherbot commented Feb 20, 2024

kkHAIKE commented Mar 1, 2024

gopherbot commented Mar 1, 2024

islinwb commented Jan 9, 2023 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

islinwb commented Jan 10, 2023 •

edited

bcmills commented Jan 10, 2023 •

edited