On https://farmer.golang.org/try?commit=c29ef957, I'm seeing multiple misc-compile TryBots (in this case, mips, netbsd, openbsd) taking upwards of 10m, when all of the real (sharded test) builders are already long done.

CC @dmitshur @toothrot @cagedmantis

On https://farmer.golang.org/try?commit=5b2140c8, I'm again seeing misc-compile TryBots taking upwards of 10m.

I too have observed this:

When these trybots take the longest, they become a bottleneck for overall TryBot completion speed (e.g., #17104).

The "misc-compile" trybots do much less work than normal trybots: they check only that compilation is successful, but don't execute tests. They're generally split by OS, matching all architectures available:

https://github.com/golang/build/blob/f539cd3e8d00b786b2c500b824a0d69d196ea3c6/dashboard/builders.go#L1600-L1609

Those architectures are currently tested sequentially. So one possible explanation is if newly added architectures start to get matched, and there become too many for sequential testing to be viable.

If that's the case, we can fix this by adjusting the split to avoid putting too much work into any single trybot.

I observed a case where misc-compile-openbsd was the slowest run:

Looking at its logs confirmed it was building as many as 5 different architectures, all sequentially:

### Building openbsd-386

### Building openbsd-amd64

### Building openbsd-arm

### Building openbsd-arm64

### Building openbsd-mips64

So the root problem here is pretty clear: we need to parallelize the workload better for OSes that have many architectures.

A fix can look like this:

-addMiscCompile("-openbsd", "^openbsd-")                 // 4: amd64, 386, arm, arm64
+addMiscCompile("-openbsd-amd", "^openbsd-(386|amd64)$") // 2: 386, amd64
+addMiscCompile("-openbsd-arm", "^openbsd-(arm|arm64)$") // 2: arm, arm64
+addMiscCompile("-openbsd-mips", "^openbsd-(mips64)$")   // 1: mips64

(We have good test coverage for missing trybots, so we'll know if something needs updating.)

I am looking at a TryBot run where misc-compile takes longer than 15 minutes, and longer than linux-amd64-longtest.

If there is an easy fix, even if hacky, I would recommend applying it, because this is adding significant latency to everyone's development cycle.

TryBots are supposed to take at most 5 minutes, IIRC, so even running two architectures in a row would probably be too slow. (15min / 4 * 2 = 7.5min)

Change https://golang.org/cl/313210 mentions this issue: dashboard: include at most 2 ports per misc-compile TryBot

With the work done to migrate x/build to OpenCensus in CL 303669, we're closer to starting to measure and track TryBot completion time, which has been a blocker for making progress on #17104.

However, the problem here is quite clear and a simple fix is available, so I've sent CL 313210 that implements the suggestion in #32632 (comment). Our updated target time for TryBot completion time is 10 minutes (see subject of issue #17104), so going with at most 2 3 ports per misc-compile TryBot should be a good compromise until we measure holistically and find something that works even better.

Change https://golang.org/cl/313533 mentions this issue: dashboard: consider disabled TryBot as not providing test coverage