Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net: 64511+ Concurrent TCP sessions (address already in use) #6176

Closed
gopherbot opened this issue Aug 17, 2013 · 7 comments
Closed

net: 64511+ Concurrent TCP sessions (address already in use) #6176

gopherbot opened this issue Aug 17, 2013 · 7 comments

Comments

@gopherbot
Copy link

by paul@vanbrouwershaven.com:

I'm currently trying to break the 64511 concurrent tcp sessions barrier in my GO
application but I encounter some unexpected behavior.

I have optimized my kernel for high tcp concurrency as suggested by Richard Jones on his
blog back in 2008 (see metabrew.com blog below). Unfortunately it doesn't have any
effect on the number of concurrent sessions.

As you have only 64511 unprivileged ports available per IP address, I have added several
additional IP addresses to my system and made go loop though these addresses. When I use
a LocalAddr with port 0 go directly starts complaining with "address already in
use" as soon I pass the +/- 64511 connections. So I tried to loop over my ip range
in combination with a specified port for the LocalAddr. Unfortunately this doesn't
improve the number of concurrent connections.

1 Million TCP connections (Linux):
http://www.metabrew.com/article/a-million-user-comet-application-with-mochiweb-part-3

2 Million TCP connections (FreeBSD):
http://blog.whatsapp.com/index.php/2012/01/1-million-is-so-2011/

I'm running a Ubuntu Server with a 3.2 kernel and go1.1.2 linux/amd64

My connection handling code:

RETRY:

// get the firs source ip:port address in queue 
ipNext = <-iprange

// manage our on port numbers per ip to overcome the 64511 limit
host, port, _ := net.SplitHostPort(ipNext)
nextPort, _ := strconv.Atoi(port)
nextPort++
if nextPort > 65535 {
    nextPort = 1024
}
    
// add this address with the next port number back to the end of the queue
iprange <- host +":"+ strconv.Itoa(nextPort)
            
d := net.Dialer{Timeout: 1*time.Second, Deadline: time.Now().Add(2*time.Second)}
d.LocalAddr, err = net.ResolveTCPAddr("tcp4", ipNext)
if err != nil {
        log.Fatal(err)
}
dc, err := d.Dial("tcp", dst)

if err != nil && strings.Contains(err.Error(), "address already in
use") {
    goto RETRY
}

--
See also the golang-nuts thread:
https://groups.google.com/forum/#!topic/golang-nuts/Mi7QkAqP7II
@remyoudompheng
Copy link
Contributor

Comment 1:

Please show the code of a complete program.

@davecheney
Copy link
Contributor

Comment 2:

Outgoing connections also consume a port. Please show a complete working example.

Labels changed: removed go1.2maybe.

Status changed to WaitingForReply.

@gopherbot
Copy link
Author

Comment 3 by paul@vanbrouwershaven.com:

Here is a demo code that is running 60k concurrent connections with a timeout of 2
seconds.
http://play.golang.org/p/mQZ2nlUQsr
Please note that you need to increase your file descriptors as below and you a multicore
system to properly run this code!
sysctl -w fs.file-max=999999
ulimit -n `cat /proc/sys/fs/file-max`
The code will print a status output like below every 5 seconds:
2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232   || ERROR: dial tcp
10.2.149.179:80: address already in use
/tmp/count.sh contains a count on current open tcp sessions:
#!/bin/sh
netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'

@gopherbot
Copy link
Author

Comment 4 by paul@vanbrouwershaven.com:

Here is a demo code that is running 60k concurrent connections with a timeout of 2
seconds.
http://play.golang.org/p/mQZ2nlUQsr
Please note that you need to increase your file descriptors as below and a multicore
system to properly run this code!
sysctl -w fs.file-max=999999
ulimit -n `cat /proc/sys/fs/file-max`
The code will print a status output like below every 5 seconds:
2013/08/19 10:51:33 GO: 60007 || TCP: ESTABLISHED 4 SYN_SENT 28232   || ERROR: dial tcp
10.2.149.179:80: address already in use
/tmp/count.sh contains a count on current open tcp sessions:
#!/bin/sh
netstat -n | awk '/^tcp/ {t[$NF]++}END{for(state in t){print state, t[state]} }'

@gopherbot
Copy link
Author

Comment 6 by paul@vanbrouwershaven.com:

This problem is kernel related. By specifying the ports in LocalAddr you can indeed
bypass the kernel problem. The actual problem in my code was 'solved' by removing a go
channel that was counting the number of cross routine completed connections,
unfortunately connections where completing faster than the channel could process them.
Increasing the channel buffer was only delaying the problem.
While you still have to take care of the "address already in use" error as it will
happen sometime when you select a port that is used by a tcp session outside the go
program.
With the code from the first post in this issue you can make millions of concurrent tcp
connections with golang.

@gopherbot
Copy link
Author

Comment 7:

Instead of picking source addresses in the app, you should be able to configure your
routing to pick from multiple source addresses. On linux, iproute2's "ip ro add ... src
..." and "equalize" options should prove useful, or go for policy routing if you must.
It'll still pick a source address first, and then try to find a free port, so the
balancing is stochastic at best.
But this seems better to me than putting all this logic in the app.

@rsc
Copy link
Contributor

rsc commented Nov 11, 2013

Comment 8:

From comment #6 it seems like everything is working as well as it could. The kernel is
getting in the way, but there's not a lot Go can do about that.

Status changed to WorkingAsIntended.

@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants