New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
proposal: syscall: add SyscallBlock #29734
Comments
This proposal makes no mention (at least directly) of RawSyscall{,6}: https://golang.org/pkg/syscall/#RawSyscall So this means we'd end up with three pairs of functions to make a system call, right? raw, normal, and block. Part of this bug would then be to document the three ways. Currently there are zero docs on the existing two pairs. |
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
This comment has been minimized.
I would say the whole syscall package needs more documentation. |
Since I forgot about |
@dvyukov, do you have any bright ideas about how we could avoid exposing these two different kinds of syscall? That is, is there anything we could do to make the regular 'entersyscall' faster for gVisor? That would be better all around than exposing yet another kind of syscall API. If the thread is blocking for a very tiny amount of time, entersyscall is better, because the blocking work is avoided entirely. If the thread is blocking for a long amount of time, that can't be happening very often so the overhead is irrelevant. So this optimization of using entersyscallblock instead of entersyscall only applies in some limited window of how long the syscall takes: not too short, not too long. I'm skeptical that users will in general be able to predict whether their particular use case falls in that window. Overuse of entersyscallblock will make some programs slower. |
In this case, we test first using We already have the RawSyscall function which provides a hint to the runtime that the call won't block or at least won't block long. A |
I don't have any magical solutions. There also few other incremental improvements like checking thread state in sysmon. I.e. sysmon could retake P when it first sees a thread in syscall if the thread status is blocked. This may reduce latency in half. There were also several proposals to export a more flexible network polling which allows to not use goroutine per read and not preallocate read buffers. That's what netstack tries to achieve, right? |
netstack is aiming for high TCP performance with low CPU usage. We have gone to great lengths to improve both of these. If you look through the repo, you will find other hacks (e.g. netstack's sleep package). |
I'm still uncomfortable committing to long-term API made available to everyone, just for gVisor's very unique needs. gVisor already reaches into the runtime internals for some things, and that seems preferable, even if it is a little more work for gVisor to keep up. Stepping back a bit, why does gVisor avoid the runtime's poller? If there's something the runtime poller is doing badly, why don't we fix that for everyone instead of giving out tons of low-level knobs? |
Other people who decided that runtime poller is not suitable for them mentioned memory overhead of goroutine per connection. I am not sure about preallocation of read buffers (Windows allows issuing 0-sized reads, but I am not sure we support this; it's also possible to issue 1-byte read first and then read out the rest, but this becomes messy and won't work for 1-byte packets; also won't work with any standard packages as they tend to allocate buffers eagerly; but maybe standard packages are not used in these contexts anyway). Overall I think people just want raw readiness notifications delivered to the specified goroutine/channel. |
The current implementation of netpoll has some performance problems. (See golang.org/cl/78915 for more information.) For this reason, we don't allow use of netpoll in gVisor. |
To go into more detail, there are a couple reasons we avoid netpoll:
In addition, gVisor often runs in platforms (e.g., KVM) where system calls can be 2-3x more expensive than native Linux processes. This can exacerbate overhead in places like findrunnable, which behave OK natively, but degrade with more expensive syscalls. I'll see if I can dig up the numbers we found for (1) and (2). |
I unfortunately can't share the complete details, but for (2), we had a real user application running in gVisor, where netpoll consumed 10% of cycles. Skipping the netpoll check (http://golang.org/cl/78915) saved those cycles and improved the application real time runtime by 5%*. *Along with http://golang.org/cl/78538, which reduced the usleep from 8% of cycles to 2%. cc @nixprime |
Interesting, so it's not memory consumption it's latency and CPU usage. |
Dmitry says "I don't see fundamental reasons why it should be slower than your manual solution." That seems like a pretty clear signal that we should not complicate the syscall API and instead should be looking at making the poller work well enough that gVisor doesn't need to reimplement it. Let's focus on that instead, please. |
In https://github.com/google/netstack/blob/master/tcpip/link/rawfile/blockingpoll_amd64.s the authors chose to write their own copy of the
Syscall
so that they could callruntime.entersyscallblock
rather thanruntime.entersyscall
. This was reportedly done to improve TCP round trip time when using gVisor (https://github.com/google/gvisor) from 182us to 112us. Presumably this is because usingentersyscallblock
, which expects the system call to block, releases the blocking P to immediately pick up some other waiting task, rather than waiting for the system monitor thread to notice that the thread has blocked.That code is currently making this performance gain in an unsafe way, by using a
go:linkname
comment to reach into an undocumented and unsupported part of the runtime package.I propose that we provide a safe mechanism for this, by adding
syscall.SyscallBlock
(andsyscall.Syscall6Block
) which would provide a way to make a system call that is expected to block. Usingsyscall.Syscall
for a syscall that blocks remains safe; the P will be blocked until the system monitor notices. Usingsyscall.SyscallBlock
for a system call that does not block will be safe; the thread will just immediately wait for the next available P.After adding this call to the syscall package, we would then of course add it to the golang.org/x/sys packages.
We could also make this change only in the golang.org/x/sys packages. The advantage of doing so is that the syscall package is frozen (http://golang.org/s/go1.4-syscall). The disadvantage is that it would require the x/sys package to reach into the runtime package in a way that will not be visible during a build of the standard library. That choice is workable but seems to me to be slightly less good.
CC @iangudger
The text was updated successfully, but these errors were encountered: