net: reconnecting to Listen'ed and forwarded ports hangs connection #22269
Comments
bradfitz
changed the title
|
@zebrig you need to show your code. Just describing your code is not good enough. Your code might have some errors. Please show small program that demonstrates your problem. Thank you Alex |
|
@alexbrainman Here is the sample code. package main
import (
"io"
"log"
"net"
"time"
"golang.org/x/crypto/ssh"
)
var (
username = "root"
serverAddrString = "yourserver:22"
knownHostKey = "|1|xxx= ecdsa-sha2-nistp256 xxx="
pkey = []byte(`-----BEGIN RSA PRIVATE KEY-----
xxx==
-----END RSA PRIVATE KEY-----
`)
)
func main() {
//Connection settings
signer, err := ssh.ParsePrivateKey(pkey)
if err != nil {
log.Fatalln("unable to parse private key: %v", err)
}
hostKey, _, _, _, err := ssh.ParseAuthorizedKey([]byte(knownHostKey))
if err != nil {
log.Fatalln("error parsing: %v", err)
}
config := &ssh.ClientConfig{
User: username,
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
// Here we connect to a local port.
//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
localListener, err := net.Dial("tcp", "127.0.0.1:5900")
if err != nil {
log.Fatalln("net.Dial failed: %v", err)
}
defer localListener.Close()
//Connecting to server
client, err := ssh.Dial("tcp", serverAddrString, config)
//Here we open a port to listen on the remote server's loopback interface.
sshListener, err := client.Listen("tcp", "127.0.0.1:12345")
if err != nil {
log.Fatalln("unable to register tcp forward: ", err)
}
defer sshListener.Close()
//Accepting connection on remote server port
sshConn, err := sshListener.Accept()
if err == nil {
go func() {
// Portforwarding. Now I see it is wrapped by a goroutine, that is not really needed, but it doesn't change anything.
go func() {
_, err = io.Copy(localListener, sshConn)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
go func() {
_, err = io.Copy(sshConn, localListener)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
}()
} else {
log.Fatalln("ssh.listen.Accept failed: %v", err)
}
for {
time.Sleep(time.Second * 5)
}
}The code itself works. To check the status of the port on the server I run this on the remote server: Port is open, now I try to connect simply by telneting the port: (vnc was used, as it has a welcome-banner): Trying to connect again: (port is open, but no welcome-banner) Or sometimes it does like this: (ssh connection either hung or breaks) Telnet is used here for clarity of what is happening. Native client software behaves the same way, throwing out errors. The golang program is run on the windows-machine, The above-mentioned commands are run on the ssh server |
|
I have done some research. The same behavior occurs if we do not open a port (sshListener, err := client.Listen("tcp", "127.0.0.1:12345")), but Listen to a unix socket. When forwarding port with other ssh software (putty, plink), it works just fine. If somebody would give me a hint, I will do a thorough testing and investigating in any needed environment to get more needed information. I already have Wireshark dumps of good connections (with putty) and bad connections (with the above mentioned code). As far as I see, the problem is that the connection is not closed properly. The connection is left abandoned by the server too early. When the server is in CLOSE_WAIT state, the Listened port does not respond with the last needed FIN and in respond the client doesnt send the last ACK. So, the client hangs in FIN_WAIT_2 state and the server hangs in CLOSE_WAIT state. As I see, the port is left half-open and that causes the ssh connection to hang. Because CLOSE_WAIT lasts until the death of ssh process, it prevents other connections to be served. While TIME_WAIT lasts for only 2 maximum segment lifetimes and is not blocking connections if the server uses SO_REUSEADDR (OpenSSH uses SO_REUSEADDR since 5.5 version as far as I remember, but not sure) |
|
@zebrig Your code above only allows a single connect via Accept. Did you expect it to continuously accept new connections? If so, you need to add a loop. |
|
@fraenkel Thank you for a good idea. But unfortunately, still no luck, same behavior. I'm attaching a fixed code. The only difference is that the for-loop now wraps the Accept block to constantly accept new connections, as you have mentioned. package main
import (
"io"
"log"
"net"
"time"
"golang.org/x/crypto/ssh"
)
var (
username = "root"
serverAddrString = "yourserver:22"
knownHostKey = "|1|xxx= ecdsa-sha2-nistp256 xxx="
pkey = []byte(`-----BEGIN RSA PRIVATE KEY-----
xxx==
-----END RSA PRIVATE KEY-----
`)
)func main() {
//Connection settings
signer, err := ssh.ParsePrivateKey(pkey)
if err != nil {
log.Fatalln("unable to parse private key: %v", err)
}
hostKey, _, _, _, err := ssh.ParseAuthorizedKey([]byte(knownHostKey))
if err != nil {
log.Fatalln("error parsing: %v", err)
}
config := &ssh.ClientConfig{
User: username,
Auth: []ssh.AuthMethod{
ssh.PublicKeys(signer),
},
HostKeyCallback: ssh.FixedHostKey(hostKey),
}
// Here we connect to a local port.
//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
localListener, err := net.Dial("tcp", "127.0.0.1:5900")
if err != nil {
log.Fatalln("net.Dial failed: %v", err)
}
defer localListener.Close()
//Connecting to server
client, err := ssh.Dial("tcp", serverAddrString, config)
//Here we open a port to listen on the remote server's loopback interface.
sshListener, err := client.Listen("tcp", "127.0.0.1:12345")
if err != nil {
log.Fatalln("unable to register tcp forward: ", err)
}
defer sshListener.Close()
//Accepting connection on remote server port
for {
sshConn, err := sshListener.Accept()
fmt.Println("A new connection has arrived")
if err == nil {
go func() {
// Portforwarding. Now I see it is wrapped by a goroutine, that is not really needed, but it doesn't change anything.
go func() {
_, err = io.Copy(localListener, sshConn)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
go func() {
_, err = io.Copy(sshConn, localListener)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
}()
} else {
log.Fatalln("ssh.listen.Accept failed: %v", err)
}
}
} |
|
I have also tried to create not a loop, but a sequence of Accept code blocks assigning sshListener.Accept() in every block to a new variable to avoid any possible problems of reusing variables in goroutines. So the code looked like this: sshConn, err := sshListener.Accept() Thus, any new connection is assigned to a new variable. |
|
I have made more investigation and found out that it is not a problem of ssh package. It is a problem of net.Listen! I have modified the program not to make ssh connections, but to do port forwarding locally:
The first I have compiled the same program for linux (Dialed 22 port - ssh), the behavior is the same. package main
import (
"io"
"log"
"net"
)
func main() {
// Here we connect to a local port.
//5900 is used for testing, because it has a welcome-banner. 445, 3389 work the same.
localDialer, err := net.Dial("tcp", "127.0.0.1:5900")
if err != nil {
log.Fatalln("net.Dial failed: %v", err)
}
defer localDialer.Close()
//Open a local port to Listen
localListener, err := net.Listen("tcp", "127.0.0.1:12345")
if err != nil {
log.Fatalln("unable to register tcp forward: ", err)
}
defer localListener.Close()
//Accepting connection on Listen'ed port to forward it to Dial'ed port
for {
localConn, err := localListener.Accept()
if err == nil {
go func() {
_, err = io.Copy(localDialer, localConn)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
go func() {
_, err = io.Copy(localConn, localDialer)
if err != nil {
log.Fatalln("%T %v", err, err)
}
}()
} else {
log.Fatalln("listen.Accept failed: %v", err)
}
}
} |
zebrig
changed the title
|
@zebrig My quick glance says your localDialer doesn't work the way you think. You are reusing it across connections which is probably a bad idea. You should be able to debug this situations but getting a goroutine dump and examining where all your routines are blocked. It is also a bit difficult when a self contained example is not provided. |
|
@fraenkel Thank you very much for your comment. You are absolutely right. For each new Listen'ed net.Conn, there should be a new Dial'ed net.Conn. I should have posted a question on stackoverflow or something before opening an issue. I'm sorry for wasting your time and thank you very very much for your help. |
What version of Go are you using (
go version)?go1.9.1 windows/amd64
What operating system and processor architecture are you using (
go env)?set GOARCH=amd64
set GOBIN=
set GOEXE=.exe
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOOS=windows
set GOPATH=C:\Users\User\go
set GORACE=
set GOROOT=C:\Go
set GOTOOLDIR=C:\Go\pkg\tool\windows_amd64
set GCCGO=gccgo
set CC=gcc
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0
set CXX=g++
set CGO_ENABLED=1
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
What did you do?
Trying to make simple port forwarding
sometimes on server side we have CLOSE_WAIT on this connection.
(if we do the same thing by putty or something, it comes to TIME_WAIT)
What did you expect to see?
Working port forward on second connect
What did you see instead?
One of three:
which is giong to happen seems to depend on traffic activity and direction on disconnect.
The text was updated successfully, but these errors were encountered: