I usually use https://golang.org/pkg/testing/#B.ResetTimer after setup code and before starting the benchmark loop. Does that work better for your use case?

I can't think of a scenario where this kind of per-size setup would lead to valid benchmark results. You're kind of lucky to be getting a timeout - the alternative would be completely bogus results.

If the setup for a loop of length b.N takes longer than the thing you're timing, benchmarks are going to have a hard time getting a precise read. The implication is that the actual operation you are doing is changing based on b.N, which invalidates the entire benchmark timing computation: a buffer of 1 GB is going to have very different cache performance than a buffer of 1 MB, but the benchmark routine depends fundamentally on b.N=1<<30 doing exactly the same operation as b.N=1<<20, just 1<<10 more times.

@rsc It's NOT per-size setup. It's what happened (which should not). I was merely trying to measure read and write performance separately. The current benchmark prediction algorithm was not able to handle it correctly (when expansive operations are ex luded).

I revised the code comment in description, it should be more intuitive now.

There are two problems here.

The first problem is that if read is cheap, then testing needs to do more of them in the loop to get an accurate per-operation iteration count. What you are suggesting would choose a very small N in the Read benchmark because of the (untimed) expensive writes, but then the calculation (timed span) / b.N would be much less accurate than usual. To make this benchmark produce accurate results, you need to find a way to move the expensive setup out of the benchmark entirely.

The second problem is that, despite the claims to the contrary, this is absolutely per-size setup. If you do N ExpensiveWrite followed by N Reads, then between those two there is some buffer somewhere containing an amount of memory that scales with N. That is exactly what you can't do in benchmarks, because the work involved in storing the memory will have different characteristics for different N. To make this benchmark produce accurate results, the actual work has to be the same, just repeated N times.

A more accurate benchmark would look like:

var initBuf = ...set up buffer contents with 1000 writes...

func BenchmarkRead(b *testing.B) {
	var buf Buffer
	for i := 0; i < b.N; i++ {
		if i%1000 == 0 {
			buf.Reset(initBuf)
		}
		buf.CheapRead()
	}
}

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

No change in consensus, so declined.
— rsc for the proposal review group