net: DNS address resolution quirks (AAAA records inconsistency) #25321

gdm85 · 2018-05-09T23:14:31Z

This bug reports is about an inconsistency on how resolution is handled between the Go resolver and the CGO one.

I do not expect a bugfix (although probably beneficial, but I leave that estimation to others) but at least an understanding of why the Go resolver behaves this way.

What version of Go are you using (`go version`)?

go version go1.10.1 linux/amd64

Does this issue reproduce with the latest release?

Latest release is 1.10.2 at the time of writing; not tested, by reading the release notes, nothing should have changed on the relevant code.

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="[...]"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="[...]"
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build482647299=/tmp/go-build -gno-record-gcc-switches"

Test setup

issue25321.go can be obtained from https://play.golang.org/p/kE_Unq4VvkO

IPv6 is disabled on this box; the DNS server may or may not return AAAA records (I have a toggle for that).

When AAAA answers are allowed:

$ nslookup -query=AAAA www.googleapis.com
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
www.googleapis.com	canonical name = googleapis.l.google.com.
googleapis.l.google.com	has AAAA address 2a00:1450:4001:825::200a

Authoritative answers can be found from:

When they are not allowed:

$ nslookup -query=AAAA www.googleapis.com
Server:		127.0.0.1
Address:	127.0.0.1#53

*** Can't find www.googleapis.com: No answer

But in both cases, an A query works:

$ nslookup -query=A www.googleapis.com
;; Warning: Message parser reports malformed message packet.
;; Truncated, retrying in TCP mode.
Server:		127.0.0.1
Address:	127.0.0.1#53

Non-authoritative answer:
www.googleapis.com	canonical name = googleapis.l.google.com.
Name:	googleapis.l.google.com
Address: 216.58.207.74
[... amended ...]

Test results

Reminder: IPv6 is always disabled, only the netdns resolver and the responses of the DNS are varying for the below tests.

Command	DNS returns AAAA	DNS does not return AAAA
`go run issue25321.go`	`2018/05/10 00:52:01 dial failed www.googleapis.com Get http://[2a00:1450:4001:821::200a]:80/: dial tcp [2a00:1450:4001:821::200a]:80: connect: cannot assign requested address` `2018/05/10 00:52:11 HTTP failed www.googleapis.com Get http://www.googleapis.com/: dial tcp [2a00:1450:4001:821::200a]:80: connect: cannot assign requested address`	`2018/05/10 00:43:23 dial failed www.googleapis.com lookup www.googleapis.com on 127.0.0.1:53: read udp 127.0.0.1:59700->127.0.0.1:53: i/o timeout` `2018/05/10 00:43:33 HTTP failed www.googleapis.com Get http://www.googleapis.com/: dial tcp: lookup www.googleapis.com on 127.0.0.1:53: read udp 127.0.0.1:43538->127.0.0.1:53: i/o timeout`
`GODEBUG=netdns=cgo go run issue25321.go`	`2018/05/10 00:47:23 dial failed www.googleapis.com Get http://[2a00:1450:4001:81d::200a]:80/: dial tcp [2a00:1450:4001:81d::200a]:80: connect: cannot assign requested address` `2018/05/10 00:47:23 OK www.googleapis.com 192.168.1.12:57248 -> 216.58.207.74:80`	`2018/05/10 00:43:07 OK www.googleapis.com 192.168.1.12:57156 -> 216.58.207.74:80`

Forgive the horrible representation, but there are two log lines at most in those table cells, you can see them better by copy/pasting their content.

Worthy of note: in the case of CGO resolver and AAAA answers allowed, first there is a failure (dialer) and then a success (HTTP request).

Another note: resolving www.bing.com is not affected by this problem, so problem must be related to how the records are returned from the DNS.

Expected results

The expected result for all the 4 combinations would be (since IPv6 is disabled on this box): do not try AAAA and use an A record, like on the bottom-right cell of the tests matrix.

Questions arising from this test

how is the order of answers handled? is there a preference to AAAA records somehow? (I am inclined to think so)
how could ever the resolution timeout when no AAAA is returned? this would be somehow the most serious part of the bug (if acknowledged), although it should first be determined if it is not a problem of the DNS (server-side)

Results at tip

With AAAA queries returned from server, GODEBUG=netdns=go+10 go run issue25321.go:

go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(www.googleapis.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
2018/05/24 08:15:44 dial failed www.googleapis.com Get http://[2a00:1450:4016:80c::200a]:80/: dial tcp [2a00:1450:4016:80c::200a]:80: connect: cannot assign requested address
go package net: hostLookupOrder(www.googleapis.com) = files,dns
2018/05/24 08:15:44 OK www.googleapis.com 192.168.1.12:35978 -> 172.217.22.234:80

real	0m1.824s
user	0m0.540s
sys	0m0.044s

With AAAA queries returned from server, GODEBUG=netdns=cgo+10 go run issue25321.go :

go package net: using cgo DNS resolver
go package net: hostLookupOrder(www.googleapis.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
2018/05/24 08:15:37 dial failed www.googleapis.com Get http://[2a00:1450:4016:80c::200a]:80/: dial tcp [2a00:1450:4016:80c::200a]:80: connect: cannot assign requested address
go package net: hostLookupOrder(www.googleapis.com) = cgo
2018/05/24 08:15:37 OK www.googleapis.com 192.168.1.12:39562 -> 216.58.207.138:80

real	0m1.606s
user	0m0.476s
sys	0m0.060s

With AAAA queries NOT returned from server, GODEBUG=netdns=go+10 go run issue25321.go:

go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(www.googleapis.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.google.com) = files,dns
go package net: hostLookupOrder(www.googleapis.com) = files,dns
2018/05/24 08:26:17 OK www.googleapis.com 192.168.1.12:41990 -> 172.217.20.138:80

real	0m2.090s
user	0m0.464s
sys	0m0.044s

With AAAA queries NOT returned from server, GODEBUG=netdns=cgo+10 go run issue25321.go:

go package net: using cgo DNS resolver
go package net: hostLookupOrder(www.googleapis.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.google.com) = cgo
go package net: hostLookupOrder(www.googleapis.com) = cgo
2018/05/24 08:27:06 OK www.googleapis.com 192.168.1.12:43918 -> 172.217.21.10:80

real	0m2.098s
user	0m0.504s
sys	0m0.068s

Conclusion

My conclusion is that problem is still present.

iangudger · 2018-05-25T19:15:10Z

@gdm85, it looks like there might have been two bugs.

In your original report, resolution failed when there was only an A and the Go resolver was in use. This was likely caused by the Go resolver rejecting the response containing the A record for some reason. This appears to have been fixed in your update.
In both the original report and the update, Go prefers the IPv6 address over the IPv4 address and does not fall back to the IPv4 address if IPv6 is disabled.

Does that sound right? If so, I think the problem may be a bug in Dial's fallback logic.

gdm85 · 2018-05-25T21:38:10Z

@iangudger yes, I missed that but with 65c365b the top-right scenario has indeed been fixed.

As for the 2nd bug, I have dug a bit deeper. These are the relevant sysctl params (notice the second one):

$ sysctl net.ipv6.conf.all.disable_ipv6
net.ipv6.conf.all.disable_ipv6 = 1
$ sysctl net.ipv6.bindv6only
net.ipv6.bindv6only = 0

For the records: on this (and similar) boxes no interface has any IPv6 address enabled/used.

I have patched a unit test to quickly determine what probe() is doing (it's a copy/paste from https://github.com/golang/go/blob/master/src/net/ipsock_posix.go with a few t.Log() added):

diff --git a/src/net/ipsock_test.go b/src/net/ipsock_test.go
index aede354..204cda3 100644
--- a/src/net/ipsock_test.go
+++ b/src/net/ipsock_test.go
@@ -7,6 +7,9 @@ package net
 import (
        "reflect"
        "testing"
+       "internal/poll"
+       "runtime"
+       "syscall"
 )
 
 var testInetaddr = func(ip IPAddr) Addr { return &TCPAddr{IP: ip.IP, Port: 5682, Zone: ip.Zone} }
@@ -280,3 +283,59 @@ func TestAddrListPartition(t *testing.T) {
                }
        }
 }
+
+// Probe probes IPv4, IPv6 and IPv4-mapped IPv6 communication
+// capabilities which are controlled by the IPV6_V6ONLY socket option
+// and kernel configuration.
+//
+// Should we try to use the IPv4 socket interface if we're only
+// dealing with IPv4 sockets? As long as the host system understands
+// IPv4-mapped IPv6, it's okay to pass IPv4-mapeed IPv6 addresses to
+// the IPv6 interface. That simplifies our code and is most
+// general. Unfortunately, we need to run on kernels built without
+// IPv6 support too. So probe the kernel to figure it out.
+func TestIPv6(t *testing.T) {
+       s, err := sysSocket(syscall.AF_INET, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
+       switch err {
+       case syscall.EAFNOSUPPORT, syscall.EPROTONOSUPPORT:
+       case nil:
+               poll.CloseFunc(s)
+               t.Log("p.ipv4Enabled = true")
+       }
+       var probes = []struct {
+               laddr TCPAddr
+               value int
+       }{
+               // IPv6 communication capability
+               {laddr: TCPAddr{IP: ParseIP("::1")}, value: 1},
+               // IPv4-mapped IPv6 address communication capability
+               {laddr: TCPAddr{IP: IPv4(127, 0, 0, 1)}, value: 0},
+       }
+       switch runtime.GOOS {
+       case "dragonfly", "openbsd":
+               // The latest DragonFly BSD and OpenBSD kernels don't
+               // support IPV6_V6ONLY=0. They always return an error
+               // and we don't need to probe the capability.
+               probes = probes[:1]
+       }
+       for i := range probes {
+               s, err := sysSocket(syscall.AF_INET6, syscall.SOCK_STREAM, syscall.IPPROTO_TCP)
+               if err != nil {
+                       continue
+               }
+               defer poll.CloseFunc(s)
+               syscall.SetsockoptInt(s, syscall.IPPROTO_IPV6, syscall.IPV6_V6ONLY, probes[i].value)
+               sa, err := probes[i].laddr.sockaddr(syscall.AF_INET6)
+               if err != nil {
+                       continue
+               }
+               if err := syscall.Bind(s, sa); err != nil {
+                       continue
+               }
+               if i == 0 {
+                       t.Log("p.ipv6Enabled = true")
+               } else {
+                       t.Log("p.ipv4MappedIPv6Enabled = true")
+               }
+       }
+}

Result of running this test:

go test -v -run TestIPv6 ./src/net/
=== RUN   TestIPv6
--- PASS: TestIPv6 (0.00s)
	ipsock_test.go:303: p.ipv4Enabled = true
	ipsock_test.go:338: p.ipv4MappedIPv6Enabled = true
=== RUN   TestIPv6MulticastListener
--- SKIP: TestIPv6MulticastListener (0.00s)
	listen_test.go:615: IPv6 is not supported
=== RUN   TestIPv6LinkLocalUnicastTCP
--- SKIP: TestIPv6LinkLocalUnicastTCP (0.00s)
	tcpsock_test.go:377: IPv6 is not supported
=== RUN   TestIPv6LinkLocalUnicastUDP
--- SKIP: TestIPv6LinkLocalUnicastUDP (0.00s)
	udpsock_test.go:283: IPv6 is not supported
PASS
Socket statistical information:
(inet4, stream|0x80800, tcp): opened=1 connected=0 listened=0 accepted=0 closed=1 openfailed=0 connectfailed=0 listenfailed=0 acceptfailed=0 closefailed=0
(inet6, stream|0x80800, tcp): opened=2 connected=0 listened=0 accepted=0 closed=2 openfailed=0 connectfailed=0 listenfailed=0 acceptfailed=0 closefailed=0

ok  	net	0.002s

Relevant lines:

	ipsock_test.go:303: p.ipv4Enabled = true
	ipsock_test.go:338: p.ipv4MappedIPv6Enabled = true

So in order to reproduce the bug(s), the machine must have IPv6 disabled but IPv6 can bind on IPv4.

To toggle AAAA functionality server-side (DNS) I am using unbound with a modified python filter (I got some clues from https://github.com/berstend/unbound-no-aaaa).

If needed I can reproduce the tests with Go 1.10.1 and master, however - before running further tests - I am also looking at identifying all the test matrix dimensions to eventually create some sort of tests suite.

Temptatively:

net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.all.disable_ipv6 = 0 (I have not tested this so far)
net.ipv6.bindv6only = 1 (I have not tested this so far)
net.ipv6.bindv6only = 0
DNS returns A, AAAA
DNS returns only A
using Go resolver
using CGO resolver
(add more dimensions here for other OSes if you want to go wild...only Linux tested so far)

Ideally we should be able to identify desired behaviour for all the (valid) permutations; for the records, on Linux all permutations are valid (might not be the case on Windows/BSD/etc).

Another relevant aspect is the total resolution time spent.

Edit: I notice now that the func supportsIPv6() does not seem used for the determination of DNS address to use/return, this is probably intentional but I ignore the rationale...

arno01 · 2018-10-09T09:32:00Z

Probably this is why the golang apps compiled without CGO (netdns) enabled are throwing Post https://any.com/sdk: dial tcp: lookup any.com on [::1]:53: read udp [::1]:41165->[::1]:53: read: connection refused (see the two referenced terraform and minio client issues above this post)

my-samsung-s8$ sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.bindv6only
net.ipv6.conf.all.disable_ipv6 = 0
net.ipv6.bindv6only = 0

Can the Go netdns be fixed in a way that it would be falling back to IPv4 if it fails to bind to IPv6? Or, rather, it would not even try IPv6 in the first place if it sees the net.ipv6.bindv6only = 0 or any other relevant to the IPv6 tunable?

There are more people starting to use Samsung DeX, they are going to get blocked by this issue, forced to recompile the software, rather than just directly using it.

Xjs · 2020-06-15T16:06:45Z

+1.

FTR: I experience this on the Google Kubernetes Engine, where IPv6 is generally not available, when using pure Go applications from inside the cluster and connecting to some Google Cloud endpoints with IPv6 records (such as www.googleapis.com). We receive ever so sporadic »cannot assign requested address« errors from inside the cluster, and Google Cloud Support pointed us in the direction of this issue.

Using the cgo-based resolver would require a different build configuration, which is something I would rather avoid. Parallelly, an issue has been raised in the Google Kubernetes Engine issue tracker to teach kube-dns to drop the AAAA records if IPv6 is not available inside a cluster.

anitgandhi · 2021-03-15T20:02:05Z

This has been affecting us in our environment as well, which is containerized linux/amd64 with IPv6 completely disabled:

→ sysctl net.ipv6.conf.all.disable_ipv6 net.ipv6.conf.default.disable_ipv6 net.ipv6.bindv6only
net.ipv6.conf.all.disable_ipv6 = 1
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.bindv6only = 0

For us, it occasionally manifests when calling the Lets Encrypt ACME API:

Post "https://acme-v02.api.letsencrypt.org/acme/chall-v3/REDACTED": dial tcp [2606:4700:60:0:f53d:5624:85c7:3a2c]:443: connect: cannot assign requested address

We've tried with CGO enabled and disabled at build time, with both resolvers (netgo vs cgo) at runtime, and also with/without setting FallbackDelay=-1 to disable Happy Eyeballs.

The issue persists for us, even with Go 1.16.

Debugging

Here's what I found when trying to dig in on the dial side, hopefully it helps other folks running into this:

Based on the discussion above, at first I thought it was an issue with ordering of the resolved addrs, and that the TCP dialer was not properly handling this case properly.

However, after reading through a lot of the dial{Parallel,Serial,Single} code, I've come to the conclusion that regardless of which dial path is taken (which is essentially driven by whether FallbackDelay > 0), the issue still persists. Both dialParallel and dialSerial will return the first valid connection and a nil error, regardless of the order.

To me, this implies that the Go resolver sometimes ONLY returns IPv6 addresses, when it should also return IPv4 in addition; it seems like the resolver sometimes just isn't including A records at all, leaving only the AAAA record's IPv6 address as the only option. Considering it's happening with both the netgo and cgo resolvers, it's also possible the environment's resolver was misbehaving. Perhaps this situation just happens less frequently with the CGO resolver, however. It's hard to say if the A record actually failed to resolve, or something else is causing it to not be included in the returned list.

Possible workarounds

I considered working around this by setting the dialer's Control function to simply return an error if network == "tcp6", but that essentially is the same thing as what happens now - if the IPv6 address is considered the primary/first, that error would be returned from dialParallel/dialSerial, and so the overall Dial would still fail.

The one option that I do believe works is explicitly specifying the LocalAddr field in the Dialer to an IPv4 address that is NOT 0.0.0.0, because it is used as a "hint" to filter out resolved addresses. When that field is not specified (nil) or is specified as 0.0.0.0, the dialer's resolution code exits out early and doesn't do any filtering on the resolved addresses.

Further, setting a custom Resolver with StrictErrors: true can help to bubble up any resolution errors earlier, rather than masking them and just getting a partial addrs list back.

Update - April 12, 2021

We ended up setting a custom dialer where we explicitly pass tcp4, ignoring the network string argument, and that ends up doing the same filtering behavior. Combining with a custom Resolver with StrictErrors: true, we were able to see that the underlying error was in fact a failure to resolve the A record for the domain; previously this underlying error got masked.

doctork9999 · 2021-10-21T13:22:21Z

The dns issue sometime involves multiple layers. By using a slightly modified code here, https://play.golang.org/p/4q-AeFKhj_o, uncommented the log statements and seeing no drop in neither IPv4 nor IPv6. Having a look into the http transport and it appeared to be that the network tcp is being used as the dial parameter opposing to specific version of IP network. It would result in both ipv4 and ipv6 name resolution and this test was running in Ubuntu so the CGO mode would have been using glibc to resolve the name based on the implementation of cgoLookupIPCNAME and with tcp network, not tcp4 nor tcp6, it should just pass the AF_UNSPEC to glibc getaddrinfo where it relies on the dns options to control the parallelism of version 4 and 6 resolution, with single-request set, as per the resolv.conf reference, we would clearly see if any of those version missing but didn't see anything, Go resolver also supports the single-request. Based on this, my conclusion now lean toward the latest update from @anitgandhi where it may seem related to some failure to resolve DNS record which I suspected that it might be related to DNS server itself.

root@9b148756fe57:/test# export GODEBUG=netdns=go+10
root@9b148756fe57:/test# go run test.go 
2021/10/21 23:38:58 inSeries
go package net: GODEBUG setting forcing use of Go's resolver
go package net: hostLookupOrder(www.googleapis.com) = files,dns
2021/10/21 23:38:58 dial success www.googleapis.com 172.18.0.2:42270 -> 142.250.70.234:80
go package net: hostLookupOrder(www.google.com) = files,dns
2021/10/21 23:38:58 dial success www.googleapis.com 172.18.0.2:60428 -> 142.250.70.196:80
2021/10/21 23:38:59 www.googleapis.com connected to www.google.com
2021/10/21 23:38:59 dial success www.googleapis.com 172.18.0.2:33260 -> 142.250.70.202:80
go package net: hostLookupOrder(www.google.com) = files,dns
2021/10/21 23:38:59 dial success www.googleapis.com 172.18.0.2:60432 -> 142.250.70.196:80
2021/10/21 23:38:59 www.googleapis.com connected to www.google.com
2021/10/21 23:38:59 dial success www.googleapis.com 172.18.0.2:43850 -> 142.250.70.170:80
go package net: hostLookupOrder(www.google.com) = files,dns
2021/10/21 23:39:00 dial success www.googleapis.com 172.18.0.2:60436 -> 142.250.70.196:80
2021/10/21 23:39:00 www.googleapis.com connected to www.google.com
2021/10/21 23:39:00 dial success www.googleapis.com 172.18.0.2:39000 -> 142.250.70.138:80
go package net: hostLookupOrder(www.google.com) = files,dns
2021/10/21 23:39:00 dial success www.googleapis.com 172.18.0.2:60440 -> 142.250.70.196:80
2021/10/21 23:39:01 www.googleapis.com connected to www.google.com
2021/10/21 23:39:01 dial failed www.googleapis.com Get "http://[2404:6800:4015:803::200a]:80/": dial tcp [2404:6800:4015:803::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:01 dial failed www.googleapis.com Get "http://[2404:6800:4015:802::200a]:80/": dial tcp [2404:6800:4015:802::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:01 dial failed www.googleapis.com Get "http://[2404:6800:4015:801::200a]:80/": dial tcp [2404:6800:4015:801::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:01 dial failed www.googleapis.com Get "http://[2404:6800:4015:800::200a]:80/": dial tcp [2404:6800:4015:800::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:01 inParallel
go package net: hostLookupOrder(www.googleapis.com) = files,dns
2021/10/21 23:39:01 OK www.googleapis.com 172.18.0.2:43858 -> 142.250.70.170:80

root@9b148756fe57:/test# export GODEBUG=netdns=cgo+10
root@9b148756fe57:/test# go run test.go 
2021/10/21 23:39:08 inSeries
go package net: using cgo DNS resolver
go package net: hostLookupOrder(www.googleapis.com) = cgo
2021/10/21 23:39:08 dial success www.googleapis.com 172.18.0.2:42288 -> 142.250.70.234:80
go package net: hostLookupOrder(www.google.com) = cgo
2021/10/21 23:39:08 dial success www.googleapis.com 172.18.0.2:60446 -> 142.250.70.196:80
2021/10/21 23:39:09 www.googleapis.com connected to www.google.com
2021/10/21 23:39:09 dial success www.googleapis.com 172.18.0.2:43864 -> 142.250.70.170:80
go package net: hostLookupOrder(www.google.com) = cgo
2021/10/21 23:39:09 dial success www.googleapis.com 172.18.0.2:60450 -> 142.250.70.196:80
2021/10/21 23:39:09 www.googleapis.com connected to www.google.com
2021/10/21 23:39:09 dial success www.googleapis.com 172.18.0.2:33282 -> 142.250.70.202:80
go package net: hostLookupOrder(www.google.com) = cgo
2021/10/21 23:39:10 dial success www.googleapis.com 172.18.0.2:60454 -> 142.250.70.196:80
2021/10/21 23:39:10 www.googleapis.com connected to www.google.com
2021/10/21 23:39:10 dial success www.googleapis.com 172.18.0.2:39018 -> 142.250.70.138:80
go package net: hostLookupOrder(www.google.com) = cgo
2021/10/21 23:39:10 dial success www.googleapis.com 172.18.0.2:60458 -> 142.250.70.196:80
2021/10/21 23:39:11 www.googleapis.com connected to www.google.com
2021/10/21 23:39:11 dial failed www.googleapis.com Get "http://[2404:6800:4015:802::200a]:80/": dial tcp [2404:6800:4015:802::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:11 dial failed www.googleapis.com Get "http://[2404:6800:4015:800::200a]:80/": dial tcp [2404:6800:4015:800::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:11 dial failed www.googleapis.com Get "http://[2404:6800:4015:801::200a]:80/": dial tcp [2404:6800:4015:801::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:11 dial failed www.googleapis.com Get "http://[2404:6800:4015:803::200a]:80/": dial tcp [2404:6800:4015:803::200a]:80: connect: cannot assign requested address
2021/10/21 23:39:11 inParallel
go package net: hostLookupOrder(www.googleapis.com) = cgo
2021/10/21 23:39:11 OK www.googleapis.com 172.18.0.2:43876 -> 142.250.70.170:80

anitgandhi · 2021-10-21T16:37:44Z

Yeah I meant to leave an update on my previous comment, that as of Go 1.17, explicitly passing tcp4 does result in the Go resolver not executing an AAAA lookup, so that was a nice optimization. It didn't really affect the tcp dialing, because tcp4 leads to IPv6 addresses getting filtered out right before the dial.

In our environment (IPv6 disabled), using tcp4 has helped a lot.

We observed intermittent failures during deployment, due to Go resolving Google API domains into IPv6 addresses, even though the Cloud Shell environment has IPv6 disabled. Until the Go issue (golang/go#25321) has been resolved, we have to patch the `/etc/hosts` file on the Cloud Shell machine to ensure that these domains are resolved using IPv4 only. PiperOrigin-RevId: 495598377

We observed intermittent failures during deployment, due to Go resolving Google API domains into IPv6 addresses, even though the Cloud Shell environment has IPv6 disabled. Until the Go issue (golang/go#25321) has been resolved, we have to patch the `/etc/hosts` file on the Cloud Shell machine to ensure that these domains are resolved using IPv4 only. PiperOrigin-RevId: 495779871

Fixes hashicorp/terraform-provider-google#6782 Workaround for golang/go#25321

agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 10, 2018

agnivade added this to the Go1.11 milestone May 10, 2018

gdm85 mentioned this issue May 10, 2018

Timing out on failed request is non-compliant m13253/dns-over-https#10

Closed

ianlancetaylor modified the milestones: Go1.11, Go1.12 Jun 27, 2018

This was referenced Oct 8, 2018

terraform vsphere provider on arm32 tries IPv6 to resolve the hostname and fails hashicorp/terraform-provider-vsphere#636

Closed

mc on arm64 tries IPv6 to resolve the hostname and fails minio/mc#2564

Closed

ianlancetaylor modified the milestones: Go1.12, Go1.13 Dec 13, 2018

eedwards-sk mentioned this issue Jul 2, 2019

Fly does not respect disabling ipv6 concourse/concourse#4083

Closed

andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019

rsc modified the milestones: Go1.14, Backlog Oct 9, 2019

mhanline mentioned this issue Jul 12, 2020

Terraform not honouring OS IPv4 settings, using IPv6 dst to call *.googleapis.com hashicorp/terraform-provider-google#6782

Open

x0b mentioned this issue Sep 17, 2020

Custom Rclone not Functioning rclone/rclone#4605

Closed

bentranter mentioned this issue Dec 15, 2020

Error when on VPN digitalocean/doctl#845

Open

Undearius mentioned this issue Apr 1, 2021

DNS lookup fails mautrix/whatsapp#291

Closed

kmoe mentioned this issue May 6, 2021

CI: GitHub Actions hashicorp/terraform-provider-dns#122

Merged

MaxKh mentioned this issue Jul 4, 2021

Unexpected requests via IPv6 rclone/rclone#5436

Closed

luisdavim mentioned this issue May 30, 2022

Linux-arm64 vs Android-arm64 cli/cli#5723

Closed

copybara-service bot mentioned this issue Dec 15, 2022

Implement a workaround for the IPv6 issue google-marketing-solutions/crmint#398

Merged

melinath added a commit to melinath/magic-modules that referenced this issue Apr 18, 2024

Made base transport layer force ipv4 usage if ipv6 is not supported

82d8a8f

Fixes hashicorp/terraform-provider-google#6782 Workaround for golang/go#25321

melinath mentioned this issue Apr 18, 2024

Made base transport layer force ipv4 usage if ipv6 is not supported GoogleCloudPlatform/magic-modules#10485

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

net: DNS address resolution quirks (AAAA records inconsistency) #25321

net: DNS address resolution quirks (AAAA records inconsistency) #25321

gdm85 commented May 9, 2018 •

edited

agnivade commented May 10, 2018

iangudger commented May 17, 2018

gdm85 commented May 24, 2018

iangudger commented May 25, 2018

gdm85 commented May 25, 2018 •

edited

arno01 commented Oct 9, 2018

Xjs commented Jun 15, 2020

anitgandhi commented Mar 15, 2021 •

edited

doctork9999 commented Oct 21, 2021 •

edited

anitgandhi commented Oct 21, 2021

net: DNS address resolution quirks (AAAA records inconsistency) #25321

net: DNS address resolution quirks (AAAA records inconsistency) #25321

Comments

gdm85 commented May 9, 2018 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

Test setup

Test results

Expected results

Questions arising from this test

Related

agnivade commented May 10, 2018

iangudger commented May 17, 2018

gdm85 commented May 24, 2018

Results at tip

Conclusion

iangudger commented May 25, 2018

gdm85 commented May 25, 2018 • edited

arno01 commented Oct 9, 2018

Xjs commented Jun 15, 2020

anitgandhi commented Mar 15, 2021 • edited

Debugging

Possible workarounds

Update - April 12, 2021

doctork9999 commented Oct 21, 2021 • edited

anitgandhi commented Oct 21, 2021

gdm85 commented May 9, 2018 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

gdm85 commented May 25, 2018 •

edited

anitgandhi commented Mar 15, 2021 •

edited

doctork9999 commented Oct 21, 2021 •

edited