New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
crypto/rand: mutex around /dev/urandom is unnecessarily broad #9221
Comments
It needs to be able to handle the case where the reader is |
If you use Go 1.4 and a modern Linux kernel we use the getrandom system call instead of open+reads of /dev/urandom. (Not that this can't be improved, but this problem will go away on its own eventually, at least on Linux...) |
Good to know, thank you, that will probably be enough to solve our immediate problem. |
I looked into this more as I was fixing #9205. The problem is that we use a bufio.Reader around the *os.File reading from /dev/urandom, and it's for the bufio.Reader that the mutex is important. If we were to fix the mutex contention issue for random, we'd probably want to do a per-processor buffer of /dev/urandom data that we tried to read from first, only falling back to re-filling the per-processor buffer from a shared (without locking) *os.File that we then do ~large reads occasionally. Then we wouldn't need bufio.Reader at all. We could use sync.Pool for the per-P buffer, but using that is contentious. |
Except on Plan9 for some reason? I'm not sure why the standard library should be buffering here at all, honestly. I understand that it will aid performance in the non-concurrent case, but my first instinct would be to let the caller do the buffering (if they so desire) and by default to just do an unlocked unbuffered Read on the underlying fd.
Most threading libraries (pthreads etc) provide a thread-local storage API that doesn't content; since go does it's own scheduler I imagine (pure speculation here) that you pin threads to processors at the OS level to avoid scheduler fights, so you could cheat and use thread-local storage to get your per-P buffers? If I'm wildly off-base feel free to ignore this :) |
I meant that it's contentious with humans, not contentious with locks. |
Oh, whoops, lol :) |
(Possibly related to #7000)
We were trying to read concurrently (on Linux, go 1.3) from the Reader provided by crypto/rand, but were seeing a lot of unexpected contention. Taking a look at the implementation, it appears to take a mutex around the entire call just in order to protect the initialization step: https://github.com/golang/go/blob/master/src/crypto/rand/rand_unix.go#L51
This seems unnecessarily broad, since the underlying
Read
call itself should be safe to do concurrently. It would seem better to replace the mutex with async.Once
which would allow concurrent reads, and also provide a fast path with an atomic op once the initialization is complete.Thoughts?
The text was updated successfully, but these errors were encountered: