Comment 1:

Seems reasonable to warn in such cases. I'm not sure I understand the proposed
criterion, though.
I'm also surprised by the outlier itself. Was there a code change there?
We'd probably want to move the implementation of -best from parsing to comparison to
accomplish this.

Comment 2:

let me rephrase the criteria more formally:
if the std. deviation of old or new data points is too large (compared
to the value themselves), then warn.
the problem is how large is too large? 100%? 50%? 30%? 10%?
PS: the data is from the m to g->m change that I mentioned in the ARM
NaCl design doc on golang-dev.

Comment 3:

Oh, so something like the coefficient of variation? Devising correct statistical tests
is way out of my depth.
As for how much is too much, I'd just naively observe a bunch of values from normal runs
and hope that the right number is obvious. And in this case, I'd say false positives are
better than false negatives.

Comment 4:

Labels changed: added release-none, removed release-go1.4.

This is a benchcmp issues that is solved by using benchstat, which is what we recommend now anyway. Closing here.