What machine and OS do you use?

OS X 10.10.2, Core i7.

To measure benchmark stability, you need to disable TuroBoost and SpeedStep in BIOS. Also bind the program to specific set of cores (for GOMAXPROCS=1 to a single core). Also make the system as idle as possible, in particular disable background processes like updaters, antiviruses, etc.
For networking benchmarks additional instability factor is kernel, and I suspect OS X is not particularly good in this respect.

On my linux machine the results are reasonably stable:

$ nice -20 taskset 16 go test -run=none -bench=BenchmarkTCP4Persistent$ -cpu=1,1,1,1,1,1,1,1 net
PASS
BenchmarkTCP4Persistent 20000 69208 ns/op
BenchmarkTCP4Persistent 20000 69545 ns/op
BenchmarkTCP4Persistent 20000 69302 ns/op
BenchmarkTCP4Persistent 20000 69450 ns/op
BenchmarkTCP4Persistent 20000 69359 ns/op
BenchmarkTCP4Persistent 20000 69095 ns/op
BenchmarkTCP4Persistent 20000 68679 ns/op
BenchmarkTCP4Persistent 20000 68962 ns/op

OneShot is more problematic, because it creates tons of TCP connections and kernel can't keep up.

In your table you say old ns/op and new ns/op. What is old and what is new.
Is old 1.3 and new 1.4 or is old 1.4 vs new is tip?

On Thu, Feb 5, 2015 at 3:33 AM, Dmitry Vyukov notifications@github.com
wrote:

To measure benchmark stability, you need to disable TuroBoost and
SpeedStep in BIOS. Also bind the program to specific set of cores (for
GOMAXPROCS=1 to a single core). Also make the system as idle as possible,
in particular disable background processes like updaters, antiviruses, etc.
For networking benchmarks additional instability factor is kernel, and I
suspect OS X is not particularly good in this respect.

On my linux machine the results are reasonably stable:

$ nice -20 taskset 16 go test -run=none -bench=BenchmarkTCP4Persistent$
-cpu=1,1,1,1,1,1,1,1 net
PASS
BenchmarkTCP4Persistent 20000 69208 ns/op
BenchmarkTCP4Persistent 20000 69545 ns/op
BenchmarkTCP4Persistent 20000 69302 ns/op
BenchmarkTCP4Persistent 20000 69450 ns/op
BenchmarkTCP4Persistent 20000 69359 ns/op
BenchmarkTCP4Persistent 20000 69095 ns/op
BenchmarkTCP4Persistent 20000 68679 ns/op
BenchmarkTCP4Persistent 20000 68962 ns/op

OneShot is more problematic, because it creates tons of TCP connections
and kernel can't keep up.

—
Reply to this email directly or view it on GitHub
#9774 (comment).

@RLH I think it's the same version. Josh just used benchcmp as a way of showing the variance between two runs (that one would hope would be nearly the same).

@dvyukov thanks for the tips. Most benchmarks are quite stable on my system; the net ones are the only ones that show significant variability. It seems plausible that this is OS X-specific and kernel-related.

Any objection to me adding b.Skip to the relevant benchmarks on darwin, since they're so noisy?

I object to adding t.Skip. It may be appropriate if the benchmark is failing episodically.
We run all benchmarks on race builders to get at least some concurrent coverage, and these look like useful benchmarks for that.
Also I run all benchmarks to measure difference in allocs/run. The benchmarks also look useful for that.

You can filter these benchmarks with -bench flag or just grep -v afterwards.

Fair enough.