Comment 1:

Note that "OSX" means OS X Mavericks and "elses OSX" means OS X Mountain Lion.

Labels changed: added os-macosx.

Comment 2:

I wonder if this is related to the 'power nap' nonsense that apple implemented in 10.9.
@Kelly, are you able to disable power nap (you can see the effect in Activity
Monitor.app) 
Does updating to 10.9.1 fix, change, make worse the problem ?

Comment 3 by kell.sommers:

I disabled Power Nap feature and didn't see any changes. As far as I know Power Nap
allows applications like email to periodically check email. It's not clear to me how
this affects performance when the machine isn't sleeping.
Also sorry for being vague on some of the details. The Mac I am testing on is already
running 10.9.1.

Comment 4 by michael.w.keesey:

FWIW, some of us have seen this under Mountain Lion which predates Power Nap.

Comment 6:

I think the attacked file explains at least part of the problem.
I don't know why it wants to read from /dev/random on every message send.

Attachments:

1. websocket.block.svg (20017 bytes)

Comment 7:

Adam, do you know if it's necessary to read /dev/random that frequently in this case?
Why?

Comment 8:

WebSockets needs to mask each message because of buggy intermediate proxies that would
otherwise have been vulnerable to cache poisoning. The masking is very dumb as I recall
- just a repeated, 32-bit XOR mask. But it does need to be random.
There are also other cases where we need very rapid entropy: AES-CBC in TLS for one,
where each record needs a random IV.
Ideally, this would be a thread-local structure (or CPU-local, where the kernel supports
it).

Comment 9:

Kelly, how important is it to you? Are you actually using mac for production?

Comment 10:

Thanks, Adam!
There is the same issue on linux as well. With GOMAXPROCS=16 with 9 seconds of execution
time, there is 550 seconds of blocking in crypto/rand.
I see that websocket reads just 4 bytes of randomness.
For a security point of view, is it a good idea to cache a little bit of /dev/random
output? I.e. read say 64 bytes, and then satisfy 4-byte requests from that buffer?
Also I am not sure why we protect reads from /dev/urandom with a mutex?...

Labels changed: added os-linux, performance.

Comment 11 by kell.sommers:

Dmitry,
I'm not using a Mac in production. 
As a developer trying to write high performance services but with only a few months of
Go experience, it threw me for a loop on my development machine. I'm not that familiar
with Go yet and I thought maybe I screwed something up and I needed to learn go routines
and channels or something because I did something wrong and it didn't feel like I was
correctly using the cores on the system somehow.

Comment 12 by kell.sommers:

How did you create that diagram? That's amazing :)

Comment 13:

Thanks, I see. Makes sense.
If you are running tests, then you just do 'go test -blockprofile=filename' and then "go
tool pprof --web binary.test filename".
It's also exposed via net/http/pprof, but you need to add
"runtime.SetBlockProfileRate(1)" in the beginning of program.
If you just running a standalone binary like in this case, then:
func main() {
    runtime.SetBlockProfileRate(1)
...
    f, _ := os.Create("/tmp/blockprof")
    pprof.Lookup("block").WriteTo(f, 2)
}
and then "go tool pprof --web binary /tmp/blockprof".

Comment 14:

Kelly: To create the file, see
$ go help testflag
in particular, the flag -test.blockprofile.

Comment 15:

@kelly, it looks like the underlying problem was traced to websockets using a lot of
random entropy and blocking on /dev/random.
What should I set the status of this ticket to ? Should it be closed ? Is there another
ticket for the underlying issue with the websocket library which this issue is blocked
on ?

Status changed to WaitingForReply.

Comment 16:

I don't understand: websocket is using crypto/rand, which should be reading from
/dev/urandom, not /dev/random. Why is /dev/urandom blocking? Or why is it using
/dev/random instead?

Comment 17:

Labels changed: added repo-net.

Comment 18:

rsc: I think it's hitting lock contention for the buffer in crypto/rand and also hitting
overhead from reading urandom.

Comment 19 by kell.sommers:

Sorry for the delay in responding! I'm a bit confused how we should proceed with this?
Haven't the results shown that performance is pretty drastic on different platforms?
I might be missing something :)

Comment 20 by ledangster:

Looks like the single Reader instance of crypto/rand is causing the issue due to the
lock being held while reading /dev/urandom (per agl's comment).
Two quick ways that this could be solved is if
(a) crypto/rand is updated to support multiple concurrent opens of /dev/urandom,
or
(b) websockets accepted a user provided random source - in which case that random source
can pool file opens to /dev/urandom
Either way, I don't think you'll be able to sort it out without support from either of
these two core components.

Comment 21 by Jens.Alfke:

AFAIK, urandom is a Linux-ism, not POSIX. The man page for urandom on OS X says:
     /dev/urandom is a compatibility nod to Linux. On Linux, /dev/urandom will
     produce lower quality output if the entropy pool drains, while
     /dev/random will prefer to block and wait for additional entropy to be
     collected.  With Yarrow, this choice and distinction is not necessary,
     and the two devices behave identically. You may use either.
So it would seem that Yarrow isn't as fast as whatever PRNG Linux's urandom switches
over to when it runs out of entropy. Makes sense.
I suggest changing the websocket code to seed itself once from the cryptographic RNG
when it opens the socket and use that seed to run a PRNG. (Which is in fact what the
regular srandomdev() and random() syscalls do on OS X.)

Unfortunately we aren’t maintaining x/net/websocket anymore and suggest alternatives at https://godoc.org/golang.org/x/net/websocket

• https://godoc.org/github.com/gorilla/websocket
• https://godoc.org/nhooyr.io/websocket