New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
net: Dial is slow on localhost #23366
Comments
How fast is it if you dial 127.0.0.1:3000 ?
… On 8 Jan 2018, at 07:05, Levin Rickert ***@***.***> wrote:
What version of Go are you using (go version)?
go version go1.9.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
What did you do?
now := time.Now()
net.Dial("tcp", "localhost:3000")
fmt.Println(time.Since(now))
This code outputs around 1s.
When replacing localhost with 127.0.0.1 it takes around 2ms.
google.de:80 is around 60ms.
Is this expected to take so much longer with localhost? Took me several hours to find out that this was the root cause of multiple issues in my application.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
Oh, sorry, you already answered this.
I don’t think there is anything that can be done, localhost is a dns name so it goes through the usual dns lookup channels.
Do you have access to a machine on a different network or with a different networking stack? Can you try to get some comparison results?
… On 8 Jan 2018, at 07:05, Levin Rickert ***@***.***> wrote:
What version of Go are you using (go version)?
go version go1.9.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (go env)?
set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
What did you do?
now := time.Now()
net.Dial("tcp", "localhost:3000")
fmt.Println(time.Since(now))
This code outputs around 1s.
When replacing localhost with 127.0.0.1 it takes around 2ms.
google.de:80 is around 60ms.
Is this expected to take so much longer with localhost? Took me several hours to find out that this was the root cause of multiple issues in my application.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
@davecheney As I said |
@davecheney Just tried under macOS, same network. 2ms for Edit: |
Ephemeral port exhaustion maybe? |
My gut says a timeout due to a dns server refusing to looking "localhost"
or balckholing the response.
…On 8 January 2018 at 08:15, Brad Fitzpatrick ***@***.***> wrote:
Ephemeral port exhaustion maybe?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#23366 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA5XFp7xaoX6v9Plis6nLjxg5ibg_ks5tITPpgaJpZM4RVxBt>
.
|
@levrik - Perhaps you can write a similar program in a different programming language and test it in the same box ? That will eliminate whether its an issue with your networking stack or with Go. |
That's a great idea. Another option, which might prove cheaper if it is
conclusive is use `ping` or something and time (with a stopwatch, unless
ping supports a one shot mode on windows) how long it takes for the first
line to appear.
…On 8 January 2018 at 15:21, Agniva De Sarker ***@***.***> wrote:
@levrik <https://github.com/levrik> - Perhaps you can write a similar
program in a different programming language and test it in the same box ?
That will eliminate whether its an issue with your networking stack or with
Go.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#23366 (comment)>, or mute
the thread
<https://github.com/notifications/unsubscribe-auth/AAAcA5Zw5oFuvyzogm8ELGrvcSbtJ3y9ks5tIZfggaJpZM4RVxBt>
.
|
|
@davecheney @agnivade Let me try later this evening. |
Sorry for not responding. |
My connections were taking exactly 1s each with
Worked around with using dialing and listening on "tcp", 127.0.0.1:port net.Listen network (tcp, tcp4) and address of ":port", "localhost:port", "127.0.0.1:port" seem affect the outcome (to my adhoc testing). For example:
go version |
I encountered the same issue, and I've done some digging. The first clue was that if I tried to connect to a port that has no listener at all the delay was 2 seconds, not 1 second. The delay isn't in the name lookup. But the results of the name lookup do matter: here, The delay arises from failed connection attempts. Specifically, the call to fd.pfd.ConnectEx takes almost exactly a second when it fails (for any address, not just local ones). You can see this by modifying the implementation to print before/after timestamps. In the The same issue affects PuTTY. I doubt that Go can do anything about this. For application developers the answer is to bind to both IPv4 and IPv6 addresses, so that it doesn't matter what connection order clients use. A couple of references:
|
Ah, that's really interesting. |
@levrik - So, would it be fair to say that this issue can be closed ? |
Timed out in state WaitingForInfo. Closing. (I am just a bot, though. Please speak up if this is a mistake or you have the requested information.) |
@agnivade Oh sorry. Completely forgot to anwser. Yes. Maybe it'd be an idea to put information about that into the Go docs? |
A PR will be much appreciated. /cc @bradfitz |
Investigation story: https://www.pivotaltracker.com/story/show/158945682 This Golang issue has the required context: golang/go#23366 (comment) - the loggregator agent only listens on localhost on ipv4 - net.Dial tries two addresses when it receives a localhost address: ::1 and 127.0.0.1 - according to the issue, the dial to ::1 fails and causes a second delay - our addition of the 1 second dial timeout and blocking dial caused us to see this on AWS and vSphere but not GCP because GCP stemcells have ipv6 disabled on all interfaces [#158991284](https://www.pivotaltracker.com/story/show/158991284)
Got same here with func main() {
ctrl.Setup()
go func() {
if err := http.ListenAndServe("localhost:3000", &poms.GZipServer{}); err != nil {
log.Fatalln("Error listening:", err)
}
}()
f, err := os.Create("cpu.prof")
if err != nil {
log.Fatalln("Can't create profiler dump file:", err)
}
pprof.StartCPUProfile(f)
for i := 0; i < 1000; i++ {
if _, err := http.Get("http://localhost:3000/api/purchaseOrders/1"); err != nil {
log.Println("Error sending request:", err)
}
}
pprof.StopCPUProfile()
} Note, that in this example both listener and client supposed to bind to same host/interface: localhost, and there is no way to get an idea of what is happening. No errors, no feedback on failed [::1] attempts, what so ever. After replacing localhost with 127.0.0.1 everything worked. I expect to receive some errors back from |
@inliquid you have a transactional race in your program. There is no guarantee that by the time your listener goroutine resolves the host part of the socket and starts the I wouldn't be surprised if changing it to the IPv4 address made this process quicker, allowing your server to listen before the clients started clienting. Another question is how you determine success. Nothing happens if the client relieves no error. I suggest creating your own listener and attaching it to the http server in two separate steps. This would allow you to implement your own |
@as well, in fact I don't have race problem. I could have such problem as you suggested, but I don't. If there were any - I would see an error as a result. I did a lot of experiments before realized what is happening, including running server and client in 2 separate processes and emulating http server to see is |
@inliquid there is not much Go can do about the connect issue. It is just reflecting the behavior of the underlying platform. The best strategies are either (1) bind to both loopback addresses, or (2) bind to just one of them but also only connect to that one. In general binding to names rather than addresses is risky: if the name resolves to multiple addresses then it will bind to only one of those addresses. There's already a warning about this under net.Listen. |
I agree in general, however, I think it's clear that localhost in Windows is exceptional case and since it has nothing to do with DNS or hosts I think it must be treated in exceptional manner, explicitly, avoiding such situations were program behavior may change unexpectedly. Maybe just make "localhost" == "127.0.0.1", or even deny "localhost" and require explicit address, smth like
|
Is this still an issue of concern, and if so do we have potential courses of action? Is this a situation where at least a note in documentation should be made for the time being? |
@EllieLockhart the root cause is in Microsoft's TCP stack and only they can fix it. There are some possible application/operator responses and they could be documented in https://golang.org/pkg/net/ (or elsewhere).
|
This comment has been minimized.
This comment has been minimized.
Thanks for the pointer @artemsnisarenko. I've submitted CL 23366 based on that. |
Change https://go.dev/cl/495875 mentions this issue: |
See #23366 (comment) for more details!
What version of Go are you using (
go version
)?go version go1.9.2 windows/amd64
Does this issue reproduce with the latest release?
yes
What operating system and processor architecture are you using (
go env
)?set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
What did you do?
This code outputs around 1s.
When replacing
localhost
with127.0.0.1
it takes around 2ms, sometimes also <1ms.google.de:80
is around 60ms.Is this expected to take so much longer with
localhost
? Took me several hours to find out that this was the root cause of multiple issues in my application.The text was updated successfully, but these errors were encountered: