Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

net: Listen("127.0.0.1:0") errors "address already in use" #16691

Closed
tamird opened this issue Aug 14, 2016 · 4 comments
Closed

net: Listen("127.0.0.1:0") errors "address already in use" #16691

tamird opened this issue Aug 14, 2016 · 4 comments
Labels
FrozenDueToAge

Comments

@tamird
Copy link
Contributor

tamird commented Aug 14, 2016 •
edited

Please answer these questions before submitting your issue. Thanks!

  1. What version of Go are you using (go version)?

    go version go1.7rc5 linux/amd64

  2. What operating system and processor architecture are you using (go env)?

    GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go- build163406223=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

  3. What did you do?
    Ran github.com/cockroachdb/cockroach/gossip's tests under stress. I haven't yet attempted to reduce the failure case.

  4. What did you expect to see?
    Success.

  5. What did you see instead?
    Since upgrading to Go 1.7 (rc3,rc4,rc5) we have been seeing the titled failures. See stress: failed test in cockroach/gossip/gossip.test: TestGossipNoForwardSelf cockroachdb/cockroach#8500 and its past incarnations: https://github.com/cockroachdb/cockroach/issues?utf8=%E2%9C%93&q=%22listen%20tcp%20127.0.0.1%3A0%3A%20listen%3A%20address%20already%20in%20use%22
@bradfitz
Copy link
Contributor

I see identical behavior for net.Listen("tcp", "127.0.0.1:0") in Go 1.6 and Go 1.7.

In both cases, I can open around 28,000 before the kernel gets angry at me:

package main  

import (
        "log" 
        "net"
)

func main() {
        var lns []net.Listener
        for i := 0; i < 1e9; i++ {
                ln, err := net.Listen("tcp", "127.0.0.1:0")
                if err != nil {
                        log.Fatalf("%d: %v", i, err)
                }
                log.Printf("%v: %v", i, ln.Addr().String())
                lns = append(lns, ln)
        }
}

Both say:

2016/08/15 18:37:31 28223: 127.0.0.1:54252
2016/08/15 18:37:31 28224: 127.0.0.1:54253
2016/08/15 18:37:31 28225: 127.0.0.1:54254
2016/08/15 18:37:31 28226: 127.0.0.1:54275
2016/08/15 18:37:31 28227: 127.0.0.1:54276
2016/08/15 18:37:31 28228: 127.0.0.1:54284
2016/08/15 18:37:31 28229: 127.0.0.1:54309
2016/08/15 18:37:31 28230: 127.0.0.1:54314
2016/08/15 18:37:31 28231: 127.0.0.1:54320
2016/08/15 18:37:31 28232: 127.0.0.1:54321
2016/08/15 18:37:31 28233: listen tcp 127.0.0.1:0: bind: address already in use

Given that you see this in a stress test, I imagine you're forgetting to close something somewhere and just running out of ports in the kernel.

I suspect the difference you see between Go 1.6 and Go 1.7 is related to GC/liveness/finalizers and you were getting luckier before in Go 1.6 with something closing listeners more aggressively for you.

I'm not sure we can help more until we know more. Ping this bug if you discover something more specific?

@tamird
Copy link
Contributor Author

tamird commented Aug 16, 2016

I've checked and we're definitely always closing our listeners in those tests.

@minux
Copy link
Member

minux commented Aug 16, 2016 via email

@tamird tamird mentioned this issue Aug 25, 2016
Closed
@tamird
Copy link
Contributor Author

tamird commented Aug 26, 2016

OK, I added ss -t at the end of a failed test. Here's what I got:

...
1534 runs so far, 0 failures, over 5s
3058 runs so far, 0 failures, over 10s
4568 runs so far, 0 failures, over 15s
6084 runs so far, 0 failures, over 20s
7577 runs so far, 0 failures, over 25s

W160826 19:47:05.370156 gossip/gossip.go:1070  not connected to cluster; use --join to specify a connected node
W160826 19:47:05.375184 gossip/gossip.go:1070  not connected to cluster; use --join to specify a connected node
--- FAIL: TestGossipNoForwardSelf (0.02s)
    client_test.go:49: listen tcp 127.0.0.1:0: bind: address already in use
FAIL
output:
State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port
ESTAB      0      0               127.0.0.1:43532            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=14))
ESTAB      0      0               127.0.0.1:45790            127.0.0.1:49686    users:(("stress.test",pid=16632,fd=26))
ESTAB      0      135             127.0.0.1:49686            127.0.0.1:45723    users:(("stress.test",pid=16632,fd=18))
ESTAB      0      0               127.0.0.1:38004            127.0.0.1:43405    users:(("stress.test",pid=16742,fd=9))
ESTAB      0      844             127.0.0.1:49686            127.0.0.1:45704    users:(("stress.test",pid=16632,fd=15))
ESTAB      0      36              127.0.0.1:38004            127.0.0.1:43427    users:(("stress.test",pid=16742,fd=15))
ESTAB      0      0               127.0.0.1:43521            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=10))
ESTAB      1737   0               127.0.0.1:60783            127.0.0.1:59332    users:(("stress.test",pid=16628,fd=36))
ESTAB      0      0              10.142.0.2:48654      169.254.169.254:http
ESTAB      0      444             127.0.0.1:59494            127.0.0.1:34639
ESTAB      0      0               127.0.0.1:41698            127.0.0.1:50167    users:(("stress.test",pid=16605,fd=12))
ESTAB      0      0               127.0.0.1:58003            127.0.0.1:43450    users:(("stress.test",pid=16687,fd=8))
ESTAB      0      0               127.0.0.1:60783            127.0.0.1:59239    users:(("stress.test",pid=16628,fd=24))
ESTAB      0      547             127.0.0.1:50167            127.0.0.1:41812
ESTAB      0      0              10.142.0.2:ssh        208.185.185.131:58167
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34619    users:(("stress.test",pid=16681,fd=24))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34583    users:(("stress.test",pid=16681,fd=15))
CLOSE-WAIT 0      0              10.142.0.2:48651      169.254.169.254:http
ESTAB      0      0               127.0.0.1:41692            127.0.0.1:50167    users:(("stress.test",pid=16605,fd=10))
ESTAB      153    233             127.0.0.1:38004            127.0.0.1:43439
ESTAB      0      0               127.0.0.1:34583            127.0.0.1:59494    users:(("stress.test",pid=16681,fd=14))
ESTAB      0      13              127.0.0.1:58035            127.0.0.1:43450    users:(("stress.test",pid=16687,fd=20))
ESTAB      0      636             127.0.0.1:56462            127.0.0.1:43529    users:(("stress.test",pid=16652,fd=13))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34572    users:(("stress.test",pid=16681,fd=11))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60162
ESTAB      0      636             127.0.0.1:56462            127.0.0.1:43521    users:(("stress.test",pid=16652,fd=11))
ESTAB      0      636             127.0.0.1:50167            127.0.0.1:41702    users:(("stress.test",pid=16605,fd=15))
ESTAB      0      0               127.0.0.1:41778            127.0.0.1:50167    users:(("stress.test",pid=16605,fd=23))
ESTAB      0      0               127.0.0.1:59263            127.0.0.1:60783    users:(("stress.test",pid=16628,fd=26))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60136    users:(("stress.test",pid=16620,fd=24))
ESTAB      0      0               127.0.0.1:34572            127.0.0.1:59494    users:(("stress.test",pid=16681,fd=10))
ESTAB      0      0               127.0.0.1:43529            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=12))
ESTAB      0      50              127.0.0.1:50167            127.0.0.1:41717    users:(("stress.test",pid=16605,fd=18))
FIN-WAIT-1 0      14              127.0.0.1:45803            127.0.0.1:49686
ESTAB      0      98              127.0.0.1:43450            127.0.0.1:58071
ESTAB      0      247             127.0.0.1:38034            127.0.0.1:39134
ESTAB      0      0               127.0.0.1:43576            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=20))
ESTAB      0      0               127.0.0.1:59170            127.0.0.1:60783    users:(("stress.test",pid=16628,fd=12))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34589    users:(("stress.test",pid=16681,fd=18))
ESTAB      0      0               127.0.0.1:56462            127.0.0.1:43552    users:(("stress.test",pid=16652,fd=18))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60102    users:(("stress.test",pid=16620,fd=21))
ESTAB      0      0               127.0.0.1:56462            127.0.0.1:43667
ESTAB      0      0               127.0.0.1:51181            127.0.0.1:39232    users:(("stress.test",pid=16753,fd=8))
ESTAB      0      0               127.0.0.1:34567            127.0.0.1:59494    users:(("stress.test",pid=16681,fd=8))
ESTAB      0      0               127.0.0.1:34578            127.0.0.1:59494    users:(("stress.test",pid=16681,fd=12))
ESTAB      0      0               127.0.0.1:56462            127.0.0.1:43639    users:(("stress.test",pid=16652,fd=27))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60043    users:(("stress.test",pid=16620,fd=9))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60053    users:(("stress.test",pid=16620,fd=11))
ESTAB      0      430             127.0.0.1:38034            127.0.0.1:39110    users:(("stress.test",pid=16751,fd=15))
ESTAB      0      0               127.0.0.1:60061            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=14))
ESTAB      0      0               127.0.0.1:43405            127.0.0.1:38004    users:(("stress.test",pid=16742,fd=8))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60088    users:(("stress.test",pid=16620,fd=18))
ESTAB      0      844             127.0.0.1:50167            127.0.0.1:41698    users:(("stress.test",pid=16605,fd=13))
ESTAB      0      13              127.0.0.1:58089            127.0.0.1:43450
ESTAB      0      13              127.0.0.1:39120            127.0.0.1:38034    users:(("stress.test",pid=16751,fd=17))
ESTAB      0      0               127.0.0.1:56462            127.0.0.1:43652
ESTAB      0      13              127.0.0.1:41832            127.0.0.1:50167
ESTAB      0      345             127.0.0.1:38034            127.0.0.1:39096    users:(("stress.test",pid=16751,fd=9))
ESTAB      0      139             127.0.0.1:51340            127.0.0.1:42405
ESTAB      0      0               127.0.0.1:50167            127.0.0.1:41858
ESTAB      0      0               127.0.0.1:39165            127.0.0.1:38034
ESTAB      0      0               127.0.0.1:51184            127.0.0.1:39232    users:(("stress.test",pid=16753,fd=10))
ESTAB      0      0               127.0.0.1:43450            127.0.0.1:58097
CLOSE-WAIT 0      0              10.142.0.2:48653      169.254.169.254:http
ESTAB      0      0              10.142.0.2:48656      169.254.169.254:http
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34691
ESTAB      0      844             127.0.0.1:50167            127.0.0.1:41692    users:(("stress.test",pid=16605,fd=11))
ESTAB      0      0               127.0.0.1:60162            127.0.0.1:36626
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34578    users:(("stress.test",pid=16681,fd=13))
ESTAB      0      0               127.0.0.1:43552            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=17))
ESTAB      0      0               127.0.0.1:39232            127.0.0.1:51192    users:(("stress.test",pid=16753,fd=15))
ESTAB      0      0               127.0.0.1:41702            127.0.0.1:50167    users:(("stress.test",pid=16605,fd=14))
ESTAB      0      430             127.0.0.1:38034            127.0.0.1:39102    users:(("stress.test",pid=16751,fd=13))
ESTAB      0      13              127.0.0.1:34669            127.0.0.1:59494
ESTAB      0      0               127.0.0.1:39232            127.0.0.1:51184    users:(("stress.test",pid=16753,fd=11))
ESTAB      0      0               127.0.0.1:58008            127.0.0.1:43450    users:(("stress.test",pid=16687,fd=10))
ESTAB      0      0               127.0.0.1:58011            127.0.0.1:43450    users:(("stress.test",pid=16687,fd=12))
ESTAB      0      0               127.0.0.1:39232            127.0.0.1:51187    users:(("stress.test",pid=16753,fd=13))
ESTAB      0      0               127.0.0.1:60053            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=10))
ESTAB      0      0               127.0.0.1:60059            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=12))
ESTAB      0      48              127.0.0.1:36626            127.0.0.1:60188
ESTAB      0      0               127.0.0.1:43499            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=8))
ESTAB      0      533             127.0.0.1:43450            127.0.0.1:58012    users:(("stress.test",pid=16687,fd=15))
ESTAB      0      0               127.0.0.1:51192            127.0.0.1:39232    users:(("stress.test",pid=16753,fd=14))
ESTAB      0      0               127.0.0.1:42405            127.0.0.1:51340
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60196
ESTAB      0      0               127.0.0.1:43685            127.0.0.1:56462
ESTAB      0      0               127.0.0.1:51187            127.0.0.1:39232    users:(("stress.test",pid=16753,fd=12))
ESTAB      0      0              10.142.0.2:48655      169.254.169.254:http
ESTAB      48     0               127.0.0.1:58103            127.0.0.1:43450
ESTAB      0      13              127.0.0.1:43652            127.0.0.1:56462
ESTAB      0      13              127.0.0.1:43439            127.0.0.1:38004
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60059    users:(("stress.test",pid=16620,fd=13))
ESTAB      0      0               127.0.0.1:34674            127.0.0.1:59494
ESTAB      0      637             127.0.0.1:43450            127.0.0.1:58011    users:(("stress.test",pid=16687,fd=13))
ESTAB      0      64              127.0.0.1:39232            127.0.0.1:51221
ESTAB      0      0               127.0.0.1:60102            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=20))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34597    users:(("stress.test",pid=16681,fd=21))
ESTAB      0      160             127.0.0.1:38004            127.0.0.1:43478
ESTAB      0      637             127.0.0.1:43450            127.0.0.1:58008    users:(("stress.test",pid=16687,fd=11))
ESTAB      0      552             127.0.0.1:43450            127.0.0.1:58003    users:(("stress.test",pid=16687,fd=9))
ESTAB      0      0               127.0.0.1:34639            127.0.0.1:59494    users:(("stress.test",pid=16681,fd=26))
ESTAB      0      0               127.0.0.1:43427            127.0.0.1:38004    users:(("stress.test",pid=16742,fd=14))
ESTAB      0      0               127.0.0.1:51340            127.0.0.1:42413
ESTAB      0      0               127.0.0.1:60654            127.0.0.1:49870
ESTAB      0      844             127.0.0.1:56462            127.0.0.1:43532    users:(("stress.test",pid=16652,fd=15))
ESTAB      0      0               127.0.0.1:60088            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=17))
ESTAB      0      13              127.0.0.1:58097            127.0.0.1:43450
ESTAB      0      0               127.0.0.1:43450            127.0.0.1:58018    users:(("stress.test",pid=16687,fd=18))
ESTAB      0      50              127.0.0.1:39232            127.0.0.1:51201
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60061    users:(("stress.test",pid=16620,fd=15))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34567    users:(("stress.test",pid=16681,fd=9))
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34669
ESTAB      0      0               127.0.0.1:43450            127.0.0.1:58035    users:(("stress.test",pid=16687,fd=21))
ESTAB      0      13              127.0.0.1:43478            127.0.0.1:38004
ESTAB      0      13              127.0.0.1:43639            127.0.0.1:56462    users:(("stress.test",pid=16652,fd=26))
ESTAB      0      0               127.0.0.1:51340            127.0.0.1:42427
ESTAB      0      0               127.0.0.1:38034            127.0.0.1:39120    users:(("stress.test",pid=16751,fd=18))
ESTAB      0      0               127.0.0.1:36626            127.0.0.1:60170
ESTAB      0      0               127.0.0.1:59494            127.0.0.1:34674
ESTAB      0      345             127.0.0.1:39232            127.0.0.1:51181    users:(("stress.test",pid=16753,fd=9))
ESTAB      0      13              127.0.0.1:43460            127.0.0.1:38004
ESTAB      0      0               127.0.0.1:42413            127.0.0.1:51340
ESTAB      0      47              127.0.0.1:49870            127.0.0.1:60654
ESTAB      0      0               127.0.0.1:39098            127.0.0.1:38034    users:(("stress.test",pid=16751,fd=10))
ESTAB      0      0               127.0.0.1:39102            127.0.0.1:38034    users:(("stress.test",pid=16751,fd=12))
ESTAB      0      0               127.0.0.1:38004            127.0.0.1:43422    users:(("stress.test",pid=16742,fd=13))
ESTAB      0      0               127.0.0.1:60188            127.0.0.1:36626
ESTAB      0      341             127.0.0.1:38034            127.0.0.1:39165
ESTAB      0      48              127.0.0.1:38034            127.0.0.1:39202
ESTAB      0      13              127.0.0.1:43499            127.0.0.1:38004
ESTAB      0      0               127.0.0.1:43409            127.0.0.1:38004    users:(("stress.test",pid=16742,fd=10))
ESTAB      0      0               127.0.0.1:42409            127.0.0.1:51340
ESTAB      0      0               127.0.0.1:51221            127.0.0.1:39232
ESTAB      0      0               127.0.0.1:42427            127.0.0.1:51340
ESTAB      0      636             127.0.0.1:38034            127.0.0.1:39098    users:(("stress.test",pid=16751,fd=11))
ESTAB      0      0               127.0.0.1:42420            127.0.0.1:51340
ESTAB      0      0               127.0.0.1:39096            127.0.0.1:38034    users:(("stress.test",pid=16751,fd=8))
ESTAB      0      0               127.0.0.1:60043            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=8))
ESTAB      0      224             127.0.0.1:51340            127.0.0.1:42420
CLOSE-WAIT 0      0              10.142.0.2:48652      169.254.169.254:http
ESTAB      0      0               127.0.0.1:43677            127.0.0.1:34703
ESTAB      0      224             127.0.0.1:51340            127.0.0.1:42409
ESTAB      0      13              127.0.0.1:60170            127.0.0.1:36626
ESTAB      0      0               127.0.0.1:43422            127.0.0.1:38004    users:(("stress.test",pid=16742,fd=12))
ESTAB      0      0               127.0.0.1:39110            127.0.0.1:38034    users:(("stress.test",pid=16751,fd=14))
ESTAB      0      13              127.0.0.1:60136            127.0.0.1:36626    users:(("stress.test",pid=16620,fd=23))
ESTAB      0      0               127.0.0.1:49870            127.0.0.1:60657
ESTAB      0      636             127.0.0.1:38004            127.0.0.1:43409    users:(("stress.test",pid=16742,fd=11))
ESTAB      0      0               127.0.0.1:39134            127.0.0.1:38034

Interestingly I'm able to reproduce this on Go 1.6 now as well.

@golang golang locked and limited conversation to collaborators Aug 26, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@bradfitz @minux @tamird @gopherbot