/cc @bradfitz

/cc @ianlancetaylor

I have marked this as a proposal for it to get the proposal treatment.

What do you actually want to do with the file descriptor?

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

It is useful to get the underlying connections file descriptor so that it can be passed into a syscall for epoll. This removes the needed to start a go routine for every websocket connection.

In my concurrent-websocket module I need to do just that so multiple websocket connections can be multiplexed through a pool of goroutines. I am currently working around this issue by using reflect and unsafe, which is really bad!

L26 below shows how I am getting the connection, and you can see on L45 how I pass that connection to netpoll.HandleReadOnce. It still uses the tls.Conn for reading and writing, but needs to pass the underlying connection for epoll to work.

https://github.com/blakerouse/concurrent-websocket/blob/master/channel.go#L26

This is also an issue when using TLS with the new http.Server ConnContext callback, in my case I want to get syscall.TCPInfo for each connection.

http.Server{
	ConnContext: func(ctx context.Context, c net.Conn) context.Context {
		// c is a tls.Conn, with apparently no way to access the actual connection
		return
	},
}

Gopherbot, this is is still an issue.

Same issue here +1
The file descriptor is needed when working with epoll WebSocket

Friends, rather than saying “I need this”, I’m sure the maintainers would find it vastly more useful to say why you need this. Be as specific and concrete as possible. Try to answer the question if this feature were added these are the specific ways it would enable me to do X which I currently cannot do. Be specific about the X, not just X the feature, but how you would change your code to use X.

Thank you

hey, I need this because I am building a static file server and wanted to restart it gracefully without interruption

@lordspace have you investigated https://godoc.org/crypto/tls#Server?

@davecheney thanks for the link. I think I've seen it but it's still not clear to me how to get the descriptor so I can pass it to the program again when it restarts.

To use tls.Server you would already hold the net.Conn for the listening socket.

I don't understand why graceful restarts need access to the file descriptor of the tls.Conn. They are usually implemented by passing the listen (not connection) socket file descriptor, which is already possible with crypto/tls:

• start a new listener with net.ListenTCP
• hold on to the (*TCPListener).File()
• wrap the new listener with tls.NewListener
• when starting a graceful restart, pass the file descriptor
• turn the file descriptor into a listener with net.FileListener
• wrap that listener with tls.NewListener

Gracefully passing active connections instead of letting the previous process run them to completion would require not just the tls.Conn file descriptor, but also a serialization of the whole TLS connection state, which is not something we'll stabilize.

@lordspace, do you have any thoughts about @FiloSottile's reply? Did you need just the listener, or every tls.Conn? If the latter, why?

@FiloSottile @rsc Thank for providing workaround options.
Do you think I'd be able to start a new SSL/TLS listener on the same SSL port ?

This is what I have to for the graceful reload the non-tls.
There's commented code related to the SSL.

func Reload() error {
	tl, ok := listener.(*net.TCPListener)
	if !ok {
		listenerError := errors.New("listener is not tcp listener")
		Warn("gracefulRestart: Failed to launch, error: %v" + listenerError.Error())
		return listenerError
	}

	f, err := tl.File()
	if err != nil {
		return err
	}

	env := os.Environ()
	env = append(env, "STATOPT_HTTPD_RESTART=1")

	var sslf *os.File

	// @todo for now I cannot get the tls listener to be "serialized" for some reason
	if sslSrv != nil { // https server is running too
		env = append(env, "STATOPT_HTTPD_RESTART_SSL=1")

		ssltl, sslok := sslListener.(*net.TCPListener)

		if !sslok {
			listenerError := errors.New("SSL listener is not tcp listener")
			Warn("gracefulRestart: Failed to launch (ssl), error: %v" + listenerError.Error())
			return listenerError
		}

		sslf, err = ssltl.File()
		if err != nil {
			return err
		}
	}

	var args []string

	// Append all previous args
	if len(os.Args) > 1 {
		args = os.Args[1:]
	}

	cmd := exec.Command(os.Args[0], args...)
	cmd.Env = env
	cmd.Stdout = os.Stdout
	cmd.Stderr = os.Stderr

	// put socket FD at the first entry
	cmd.ExtraFiles = []*os.File{f}

	if sslf != nil {
		cmd.ExtraFiles = append(cmd.ExtraFiles, sslf)
	}

	startErr := cmd.Start()

	if startErr != nil {
		Warn("gracefulRestart: Failed to launch, error: %v" + startErr.Error())
	}

	return startErr
}

This is what I have if it's a graceful restart.

go func() {
		var err error

		if gracefulRestart {
			f := os.NewFile(3, "")
			listener, err = net.FileListener(f)
		} else {
			//err = srv.Serve(l)
			//err = srv.ListenAndServe()
			listener, err = net.Listen("tcp", srv.Addr)
		}

		if err != nil {
			log.Fatal("Cannot start listening:", err)
		}

		netListener := listener
		//		netListener := newGracefulListener(listener)
		srv.Serve(netListener)
	}()

@lordspace, thanks. That's just the listener being passed between those two programs, not the tls.Conn.

The http.Server's ListenAndServeTLS method looks like this:

func (srv *Server) ListenAndServeTLS(certFile, keyFile string) error {
	if srv.shuttingDown() {
		return ErrServerClosed
	}
	addr := srv.Addr
	if addr == "" {
		addr = ":https"
	}

	ln, err := net.Listen("tcp", addr)
	if err != nil {
		return err
	}

	defer ln.Close()

	return srv.ServeTLS(ln, certFile, keyFile)
}

Instead of calling that method you'd want to arrange to do the net.Listen yourself (on ":443") and then save that listener for use in the graceful restart. And then also pass the listener to srv.ServeTLS to serve HTTPS. And then after a graceful restart handoff you'd just call srv.ServeTLS in the restarted program.

It doesn't seem like there needs to be support in crypto/tls to enable this.

Looking at the other rationales put forward for getting the underlying fd from the tls.Conn,

@FMLS:

I need to access Fd too, In public cloud environment, we need this fd to parse VPC infomation

Can you say more about this use case?

@LeGEC:

I have some code where I would like to call (*TCPConn).CloseWrite() from a TLS connection,

Can you say more about this use case? It's unclear to me whether half-closing a fd out from under TLS is well defined. Perhaps it is?

@bradfitz:

If you want to find stuck connections it can be useful to take the TLS fd and look it up in the kernel's socket debugging tables.

@jtorvald:

It's just that with a lot of connections the limiting of go routines is really helpful to keep memory consumption to a minimum (but yeah, I almost feel uncomfortable to write that in a reply to you ;-)).

I asked above: "What are the memory constraints you have where a goroutine with minimal stack is too much overhead? It seems like we'd be talking about something like 2-4 kB per connection."

It sounds like we need more information about use cases before we can figure out what the right API is.

@rsc I rest my case. If needed I will re-open it in the future when I have some real numbers.

To me it seemed as legit as any other optimization issue raised here. I appreciate all the work on making go faster, smaller and better. I thought a tiny change here could help safe memory and fulfill other people's use cases. If it's considered to impactful or not worth it then it's fine by me. Thanks for your time.

Ping @FMLS and @LeGEC if you have anything to add in response to the questions in #29257 (comment).

Not retitling (yet), but worth pointing out that not every tls.Conn has a file descriptor, so really this would be about allowing a tls.Conn to return the underlying net.Conn.

Our use case :

We wrote a proxy, which connects to our server, and allows us to multiplex individual connections (conn1, conn2 in the diagram) to external service(s) (service1, service2) over a single real connection (realConn).

  Server (ours)                       Proxy (ours)     |  Service (not ours)
+--------------+                    +--------------+   |
|              |                    |              |   |
|   conn1 -*   |      realConn      |   *- prox1 -------- service1
|    (tls)  \  +--------------------+  /   (blind) |   |   (tls)
|            --------------------------            |   |
|            --------------------------            |   |
|           /  +--------------------+  \           |   |
|   conn2 -*   |                    |   *- prox2 -------- service2
|    (tls)     |                    |      (blind) |   |    (tls)
+--------------+                    +--------------+   |

We wrote a simple (simplish ?) protocol where each packet of data gets sent over realConn along with an id that identifies the connection, and also allows to exchange "out of band" messages.
For example :

• the Proxy can send a message stating "when I tried to write on connection prox2, I got the following error : ...",
• the Server can send a message "close connection prox1, I won't use it anymore".

Our proxy is generic, and doesn't understand the protocol that transits over the connection ; when connections to the Service require TLS, our proxy is completely "blind" regarding the state of the TLS connection.

Our "out of band" messages are important for bookkeeping : that's what allows us to control that connections and resources are actually released at some point (on the Server side as well as on the Proxy side).

Connection to this issue :

• I first wanted to test what happened if I called .CloseWrite() on the TCP connection of a regular TLS connection, and realized it wasn't as straightforward as I thought ;
• in our actual use case, the underlying connection is not a net.TCPConn, it is a custom ourpkg.ProxiedConn ;
• our more generic need is to be able to send "out of band" messages to our Proxy on this connection.

A sketch of implementation I had in mind was : in some use cases, call something like .CloseWrite(), to indicate that the Server will not send any more information on that connection, but keep the reading end open to see if more information come downstream.

After further thoughts, and reading the other comments in this thread, I realize calling shutdown(fd, SHUT_WR) at the TCP level of a TLS connection is perhaps not a great idea ;

but to fulfill our need to send "out of band" messages to our proxy, we have to keep together in a struct both the "clear" connection and the tls connection, in places where accessing the underlying net.Conn (exactly as @rsc pointed out in his last comment) would seem pretty natural.

one extra point : this is not a show stopper, since we can implement what we want, in our case it would be more of a nice to have.

Thanks for the details @LeGEC. If we're to commit to exposing the underlying net.Conn, it would be best to have a setting where that's strictly required, as opposed to just "a little nicer".

@FMLS, can you say more about the Cloud VPC usage?

In November, @FMLS wrote:

I need to access Fd too, In public cloud environment, we need this fd to parse VPC infomation

Does anyone know what public cloud APIs work in terms of file descriptors (which would potentially justify accepting this proposal)? Thanks.

I just came across this and wanted to share my experience. Forgive me for reiterating on something which has already been discussed, but I want to go back to the websockets and fd use case.

My use case was similar to OP. I wanted to revamp the websocket architecture for Mattermost using a single epoll instance and get rid of all the reader goroutines. I actually don't require full access to net.Conn, just being able to get the fd so that I can stick it into the epoll instance should be good enough for me.

Epoll is not a good enough reason to expose the file descriptor.

Personally, it is a matter of API consistency to me. TCPConn, UDPConn, IPConn all of them have the File method. I would expect tls.Conn to have it too.

worth pointing out that not every tls.Conn has a file descriptor, so really this would be about allowing a tls.Conn to return the underlying net.Conn.

Fair point. In that case, I would expect it to return a nil.

What are the memory constraints you have where a goroutine with minimal stack is too much overhead? It seems like we'd be talking about something like 2-4 kB per connection.

The argument is similar to the fact that while goroutines are cheap, they are not free. Having 100,000 connections (which is not that implausible) costs 400MB, which starts to matter in a multi-tenant Cloud environment.

But more importantly, it feels a bit weird to me that we can have epoll instances for HTTP servers, but not HTTPS servers (without resorting to hacks). If the argument is that goroutines are cheap, and we should let the runtime do epoll, it should apply to TCPConn as well.

EDIT: Just realized another use-case that I have. I want to call SetNoDelay on a TLS websocket connection. That's also not possible in the current state of things.

SetNoDelay and maybe SetKeepAlive seem like the strongest arguments so far.
If we just provide access to the net.Conn then people can use the net.Conn to get an fd if they really need it.
This seems OK.

Does anyone object to adding

package tls
func (c *Conn) NetConn() net.Conn

?

/cc @FiloSottile

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

@rsc Thank you for fixing this, when will this issue be resolved?

Change https://golang.org/cl/325250 mentions this issue: crypto/tls: add NetConn method