Please don't post screenshots, can't copy paste from them, have links to commits, codeblocks and other textual form, ...

We really need more repeated runs of each binary to be statistically analyzed, for all that I know adding or removing empty print lines has negligeable effect and you are seeing random variance which would also happen if you built a binary once and ran it 5 times.
We have the benchstat tool which compute that information from the go standard testing.B output.

We've seen this before, performance varying a lot based on how code is laid out in memory. We do try some things to mitigate this, like aligning various pieces of code.
@drchase has been doing some more investigations about good/bad alignments. See CL 576835 and CL 577935.

TL;DR If this effect is what I think it is, there's not much the Go project can do. You'll have to get your CPU vendor to stop making chips that have random(ish) performance variation.

@Jorropo Thank you for pointing out the benchstat tool! I'll give it a try!

@randall77 Thank you for pointing out that it could be CPU related. I've run some tests on 4 different PCs after I've read your post.

1. My office PC, an AMD 5950x
2. My current gaming PC, an AMD 5700x
3. My previous gaming PC, an Intel i5-4690K
4. My Raspberry Pi 4B

• 5950x and 5700x both have the exact same behavior. When performance good, it's good on both, but when it's bad, it's also bad on both (no matter if it was compiled by AMD, Intel or RPI)
• i5-4690K is very consistent! I can't say if performance is optimal, but it's always the same!
• Raspberry Pi 4B seem to have a small variance but it's not even close to be as bad as my 2 AMD CPU. I would say that the variance seems to be within margin of error but it's not as good as my intel CPU.

I'm having a hard time convincing myself that it's AMD's fault since I was making the benchmark in C before switching to Go and I have never had this problem with the C implementation. Also, simply adding a line fmt.Print() in the code can worsen or improve performance by as much as 32% which is a very big performance degradation.

I'll continue investigate this problem for a bit and I will open a new issue or even a PR but only if I come to find something tangible.

By the way, goroutines are awesome! ❤️ Thank you!

Cheers!
Dave

I don't mean to be annoying. I make this last post for the sake of completeness.

Output of go test -bench=BenchmarkMain -count=8 > old.txt:

goos: linux
goarch: amd64
pkg: github.com/PerfectlyFluffy/perfect28
cpu: AMD Ryzen 9 5950X 16-Core Processor            
BenchmarkMain-32    	       1	62698387947 ns/op
BenchmarkMain-32    	       1	63401121520 ns/op
BenchmarkMain-32    	       1	63459067340 ns/op
BenchmarkMain-32    	       1	63851727405 ns/op
BenchmarkMain-32    	       1	63563447685 ns/op
BenchmarkMain-32    	       1	63841900151 ns/op
BenchmarkMain-32    	       1	64007010843 ns/op
BenchmarkMain-32    	       1	64087389162 ns/op
PASS
ok  	github.com/PerfectlyFluffy/perfect28	508.915s

Output of go test -bench=BenchmarkMain -count=8 > new.txt (only difference in the code versus old is a fmt.Print() at the root of the main function):

goos: linux
goarch: amd64
pkg: github.com/PerfectlyFluffy/perfect28
cpu: AMD Ryzen 9 5950X 16-Core Processor            
BenchmarkMain-32    	       1	81852111106 ns/op
BenchmarkMain-32    	       1	81963111956 ns/op
BenchmarkMain-32    	       1	82237281288 ns/op
BenchmarkMain-32    	       1	82279110930 ns/op
BenchmarkMain-32    	       1	82491095339 ns/op
BenchmarkMain-32    	       1	82303076903 ns/op
BenchmarkMain-32    	       1	82481060072 ns/op
BenchmarkMain-32    	       1	82489785819 ns/op
PASS
ok  	github.com/PerfectlyFluffy/perfect28	658.101s

Output of benchstat old.txt new.txt:

goos: linux
goarch: amd64
pkg: github.com/PerfectlyFluffy/perfect28
cpu: AMD Ryzen 9 5950X 16-Core Processor            
        │  old.txt   │              new.txt              │
        │   sec/op   │   sec/op    vs base               │
Main-32   63.70 ± 2%   82.29 ± 1%  +29.18% (p=0.000 n=8)

Here a screenshot of the command I used to run the benchmark and the output it yield:

Explanation of the command:

1. Preheated the CPU to mitigate cold start advantage.
2. Used sed to comment the fmt.Print() in the main to set the benchmark in "good performance mode".
3. Used git diff to show that sed had done the job properly and also to show that the folder is not dirty.
4. Ran 8 loop of BenchmarkMain and output the result into old.txt
5. Used sed again to uncomment the line fmt.Print() to set the benchmark in "bad performance mode".
6. Used git diff again for the same reason that point 3.
7. Ran 8 loop of BenchmarkMain and output the result into new.txt

Sadly, I could not include the execution of benchstat old.txt new.txt into the command because I first had to remove manually the output of my program.

I noticed that any code I put in my BenchmarkMain function never affects performance. I've try in many ways, but performance always stayed the same. Only code use in the Main function seem to have an effect on performance.

I provided the benchmark function and instructions to reproduce in this commit: PerfectlyFluffy/perfect28@7e84474

I think this is the best I can do to help with this issue. At this point it appears to be obvious that I did all of this for nothing, but I had a lot of fun.

Cheers,
Dave