Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: net: add an option to provide the socket filedescriptor to the connection #46932

Closed
aojea opened this issue Jun 25, 2021 · 9 comments
Closed
Labels
FrozenDueToAge Proposal WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@aojea
Copy link
Contributor

aojea commented Jun 25, 2021

Golang and network namespaces have an "interesting" history #44922, basically mixing goroutines and network namespaces is a bad idea.

However, there was an interesting discussion in the kernel mailing list back in 2010 to

Add a system call for creating sockets in a specified network namespace.

the conclusion is that the same functionality should be easy to implement in userspace, so no need to add a new syscall

Strictly speaking with setns() you can implement this functionality
with setns(). aka

int socketat(int nsfd, int domain, int type, int protocol)
{
        int sk;

        setns(0, nsfd);
        sk = socket(domain, type, protocol);
        setns(0, default_nsfd);

        return sk;
}

I think that this can solve some of the problems of golang with network namespace, allowing to create a socket inside a namespace, but using it outside of it, so no need to lock the thread and remove the limitation of creating additional goroutines.

It seems that right now we can manipulate the socket but not the file descriptor

go/src/net/sock_posix.go

Lines 17 to 23 in ed01cea

// socket returns a network file descriptor that is ready for
// asynchronous I/O using the network poller.
func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {
s, err := sysSocket(family, sotype, proto)
if err != nil {
return nil, err
}

What is the golang opinion about this?
Is this something interesting to add to the language?

@seankhliao seankhliao changed the title net: Implement socketat https://lore.kernel.org/patchwork/patch/217025/ proposal: net: add socketat for network namespaces Jun 26, 2021
@gopherbot gopherbot added this to the Proposal milestone Jun 26, 2021
@seankhliao
Copy link
Member

Are the concerns there addressed?

The major difference is that socketat in userspace suffers from races, with signals etc.

@ianlancetaylor
Copy link
Contributor

It's hard to be sure, but this looks like the kind of thing intended to be addressed by code like TCPConn.SyscallConn to get a syscall.RawConn and then call the Control method.

If that doesn't help, can you explain what kind of thing would help? Right now this is a little vague. Thanks.

@ianlancetaylor ianlancetaylor added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 27, 2021
@aojea
Copy link
Contributor Author

aojea commented Jun 27, 2021

Are the concerns there addressed?

The major difference is that socketat in userspace suffers from races, with signals etc.

I think that this is in the context of using C, I expect golang and its runtime to handle this , the SocketAt func will be something like this:

func SocketAt(domain, typ, proto, ns int) (int, error) {
	// get current namespace
	origin, err := os.Open("/proc/self/ns/net")
	if err != nil {
		return -1, err
	}
	// lock the thread so we don't switch namespaces
	runtime.LockOSThread()
	defer runtime.UnlockOSThread()
	// enter the new namespace
	err = unix.Setns(int(ns), syscall.CLONE_NEWNET)
	if err != nil {
		return -1, err
	}
	// always come back to the original namespace
	defer func() {
		if e := unix.Setns(int(origin.Fd()), syscall.CLONE_NEWNET); e != nil {
			log.Printf("failed to recover netns: %+v", e)
		}
	}()
	// open the socket in the new namespace and return its file descriptor
	return syscall.Socket(domain, typ, proto)
}

If that doesn't help, can you explain what kind of thing would help? Right now this is a little vague. Thanks.

yeah, let me elaborate and try to explain it more on detail, bear in mind this is too low level to me and I may be doing some false assumptions, so please correct me if I'm wrong.

Right now, if I want to listen or dial from a namespace I have to lock the thread, enter the namespace and do the network operations there, no further goroutines allowed inside the namespace, ... more details on https://www.weave.works/blog/linux-namespaces-golang-followup

Let's say that I want to have an http server in each namespace/container/pod (serving the same content in all of them), or that I need to create connections from the namespaces, everything handled by a single process in the root namespace.
AFAIK the only solutions that I have today are, from the main process in the root namespace :

  1. fork/exec and create a new process per namespace
  2. create a goroutine per namespace with the thread locked.

Both solutions seems to have scale/performance problems on environments with large number of namespaces, order of 100s.
I think that if I'm able to get the socket from the namespace, I can avoid the limitation to lock the thread, and handle the network namespaced operations from the main process using gorotuines.

but this looks like the kind of thing intended to be addressed by code like TCPConn.SyscallConn to get a syscall.RawConn and then call the Control method.

I was looking at that and also reading the discussion about the Control method, but I can't see how can I leverage that to switch to the namespace, create the socket and return to the original namespace ... it seems that Control applies after the file descriptor has been created.

Hope this clarifies a bit more the problem ...

@ianlancetaylor
Copy link
Contributor

As far as I can tell you can write the SocketAt function already without requiring any new features. Also, since it uses the golang.org/x/sys/unix package, it can't be in the standard library as-is. So I guess the question is whether there is any advantage to adding the function to golang.org/x/sys/unix. Do we expect lots of people to need this function? Are there other similar functions, or is there something special about sockets? Thanks.

@aojea
Copy link
Contributor Author

aojea commented Jun 27, 2021

So I guess the question is whether there is any advantage to adding the function to golang.org/x/sys/unix. Do we expect lots of people to need this function? Are there other similar functions, or is there something special about sockets? Thanks.

I can't see any advantage of adding a SocketAt to golang.org/x/sys/unix and I start to think that this must be the conclusiong that kernel developers reached. Acually, I can only see this use case because of the golang limitation with network sockets, namespaces and gorutines.

I've wrapped current net.Dial and net.Listen with the SocketAt code to create net.DialAt and net.ListenAt , with the code I pasted above and it does work: https://github.com/aojea/socketat/blob/main/socketat_linux_test.go, it manages the network connections on different namespaces from the default namespace,
However, that implementation seems to brittle to me, the Dial and Listen method has much surface and I don't think that will be race free ... If I just could wrap the socket creation to do the namespace dance ...

go/src/net/sock_posix.go

Lines 19 to 23 in f448cb8

func socket(ctx context.Context, net string, family, sotype, proto int, ipv6only bool, laddr, raddr sockaddr, ctrlFn func(string, string, syscall.RawConn) error) (fd *netFD, err error) {
s, err := sysSocket(family, sotype, proto)
if err != nil {
return nil, err
}

I don't think that the socketat code should go in golang either, the only solutions I have in mind are if:

  • I have a way to provide the filedescriptor to the Connection?
  • I can execute a function, as in Control, to influece the socket creation?

@aojea aojea changed the title proposal: net: add socketat for network namespaces proposal: net: add an option to provide the socket filedescriptor to the connection Jun 28, 2021
@aojea
Copy link
Contributor Author

aojea commented Jun 28, 2021

  • I have a way to provide the filedescriptor to the Connection?

This is definitevely the solution I'd like to have, being able to create a network connection from a socket file descriptor I've created ... I don't know if this is too crazy though 😅

@ianlancetaylor
Copy link
Contributor

To create a socket yourself and then turn it into a net.Conn, use os.NewFile and net.FileConn.

@ianlancetaylor
Copy link
Contributor

As far as I can tell, there isn't anything to change in the Go project here. Please comment if you disagree.

@aojea
Copy link
Contributor Author

aojea commented Jul 14, 2021

As far as I can tell, there isn't anything to change in the Go project here. Please comment if you disagree.

sorry I forgot to close it, indeed there is nothing about Go here

Thanks

@golang golang locked and limited conversation to collaborators Jul 14, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge Proposal WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants