TestGcSys has been flaky, see #37331 . Is there any other failure other than this one?

I doubt this is PPC64-specific.

This fails consistently on a system with > 400 processors. That means by default ./all.bash fails on that system.
You are correct, I can make it fail consistently on an x86 if I set GOMAXPROCS=400 so it is not PPC64 specific.

cc @aclements @mknyszek

@laboger do you see any other test failing? TestGcSys is a known flaky test.

I'm still looking.... Some are taking much longer with more GOMAXPROCS, so I'm trying to determine if they are hanging.
(An earlier run had failures/panic traces but I didn't capture that output.)

Here is a test that timed out so I wasn't sure if it was a hang, turns out it just takes a lot longer as the GOMAXPROCS increase. This system has 448. Not sure if that is expected behavior for the test.

[boger@rain6p1 runtime]$ GOMAXPROCS=64 go test -test.run=TestSemaHandoff
PASS
ok  	runtime	10.074s
[boger@rain6p1 runtime]$ GOMAXPROCS=128 go test -test.run=TestSemaHandoff
PASS
ok  	runtime	79.771s
[boger@rain6p1 runtime]$ GOMAXPROCS=245 go test -test.run=TestSemaHandoff
PASS
ok  	runtime	292.193s
[boger@rain6p1 runtime]$ go test -test.run=TestSemaHandoff -test.timeout=30m
PASS
ok  	runtime	977.224s

@laboger Oof. I do think TestSemaHandoff is going to scale pretty poorly with that many processors, if I'm remembering that test correctly. Maybe it should have a cap on GOMAXPROCS.

I know for certain that the runtime is going to have some scalability issues at that many processors. The scheduler and allocator don't take advantage of NUMA at all, for instance.

As Cherry points out, TestGcSys is unfortunately just known to be flaky, and theoretically it's independent of GOMAXPROCS (it sets its own GOMAXPROCS to 1). It could just be that the specific scheduling that triggers the GC issues causing the flakiness is just more common on your system. I'm not sure.

I understand, I'm just suggesting that the default for GOMAXPROCS could have an upper limit instead of always basing it on the number of processors on the system, since there known issues and performance hits above a certain point.

Talking with the team, the consensus was that we should just limit GOMAXPROCS in the tests that don't scale well. @prattmic has done benchmarks that show Go itself can scale pretty well to 512 and even 1024 GOMAXPROCS. This is obviously pretty workload-dependent, so I don't think we want to artificially limit GOMAXPROCS. But some of the tests just don't scale well, and we can keep those from timing out.

I understand, I'm just suggesting that the default for GOMAXPROCS could have an upper limit instead of always basing it on the number of processors on the system, since there known issues and performance hits above a certain point.

I definitely don't think we should cap explicitly specified values of GOMAXPROCS, however perhaps it could be OK to cap the default value of GOMAXPROCS if there are serious performance issues for real-world applications (though hopefully we could work towards fixing those). I don't think this test is a realistic workload for that purpose, but I'd be curious to see other common applications.

@mknyszek @aclements I had opened this issue back in 2021 related to GOMAXPROCS and it was interesting to hear others are hitting the problem on systems other than Power. After reviewing this I had a few thoughts after the meeting.

• Based on the discussion in the office hours, it seems that having a lower default value for GOMAXPROCS makes sense. If there are cases where someone wants to set it higher they can, but it doesn't make sense to me that the default can be a value that can cause issues or seriously degrade performance.
• If you aren't going to make a change then this behavior should be documented somewhere.

I recommend users on our big systems to use GOMAXPROCS=64.