Unfortunately I can't tell which runtime test is failing. The stack trace shows two goroutines whose stack is not reported because when the timeout occurs they are running on a different thread. It would be interesting to know if the failing test is TestSelectFairness, which was introduced in the same CL as the fastrand change.

I'm trying to get information from others who are building from upstream to see if anyone else has hit this problem. So far we've only seen it on the build page. I can't seem to find the information about the build machine, such as what distro is it and what would be the default GOMAXPROCS on that system? With that information I could see if it reproduces with the exact same environment.

The test TestSelectFairness could be skipped on ppc64le at least temporarily to see if that prevents the failure from happening and then we would be pretty sure that test introduced the problem.

Change https://golang.org/cl/67631 mentions this issue: runtime: skip test that intermittently hangs on ppc64le

Could it be due some field reordering in type m struct {} ?
https://go-review.googlesource.com/c/go/+/64193/1/src/runtime/runtime2.go

And I found place, where fastrand assumed to be non-zero:

Yes, I agree it could be something else. If we skip this test and it continues to hang then we'll know it's not the test.

Test has failed after skipping this test.

What happens with such patch:

diff --git a/src/runtime/sema.go b/src/runtime/sema.go
index 8715e07d7a..816e894ae5 100644
--- a/src/runtime/sema.go
+++ b/src/runtime/sema.go
@@ -275,7 +275,7 @@ func (root *semaRoot) queue(addr *uint32, s *sudog, lifo bool) {
        // on the ticket: s.ticket <= both s.prev.ticket and s.next.ticket.
        // https://en.wikipedia.org/wiki/Treap
        // http://faculty.washington.edu/aragon/pubs/rst89.pdf
-       s.ticket = fastrand()
+       s.ticket = fastrand() | 1
        s.parent = last
        *pt = s

???

There may well be a problem in sema.go but why would it only show up on ppc64?

Change https://golang.org/cl/68050 mentions this issue: runtime: fix using fastrand in sema.go

ppc64le build tests continue to hang intermittently.

Change https://golang.org/cl/68710 mentions this issue: runtime: reorder fields in type m

Fastrand is still guaranteed to always return non-zero values, so surely the patch that went in can’t fix anything.

I still wouldn’t revert it because I don’t think code should rely on this property, it’s confusing and fragile.

I don't think fastrand is guaranteed to return nonzero any more.
Its internal [2]uint32 state can't be all zero, but we return state[0]+state[1], which could end up zero.

Yes, xorshift64 with 32bit result returns zero easily.

This test is no longer failing.

I wish we understood exactly what fixed this, but closing anyway since it's no longer failing.

The test is no longer failing because it's now skipped on ppc64le.
Whatever is underlying is probably still happening (although I haven't checked).