Thanks for reporting this.

Coordinator finds work by using Gerrit's API, so if Gerrit is having issues, that would explain trybots not starting. We should figure out if the problem is on the side of Gerrit or not.

I see at https://farmer.golang.org/#trybots there are active trybot runs right now, so maybe this is an issue only affecting a subset of CLs?

I see at https://farmer.golang.org/#trybots there are active trybot runs right now,

They all seem to be stuck though. All builds have completed, yet the trybot run isn't completed.

From coordinator logs, it seems clear that the problem is coordinator is failing to find work:

$ kubectl logs  coordinator-deployment-6cff4d4d8d-5j6f5  | grep "failed to find trybot work:"
[...]
2020/12/21 23:37:21 failed to find trybot work: rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A
2020/12/21 23:37:36 failed to find trybot work: rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A
2020/12/21 23:37:52 failed to find trybot work: rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A
2020/12/21 23:38:06 failed to find trybot work: rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A
2020/12/21 23:38:22 failed to find trybot work: rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A

Ib1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5 is the Change-Id of the duplicate CL that was mentioned.

Coordinator will need to be updated to handle it without an error.

Is it possible to Gerrit super-user force delete https://go-review.googlesource.com/c/build/+/279515 or something? Maybe that would help unstick the situation?

Yes, deleting the CL may work.

We should also make coordinator more resilient so this won't need to be dealt with manually if a duplicate Gerrit CL happens again in the future.

I suspect the problem could be how we're specifying the ID in the call here:

comments, err := gerritc.ListChangeComments(ctx, ci.ID)
if err != nil {
	return nil, err
}

But I can't reproduce it locally with maintq try-work. Not sure why yet.

We should also make coordinator more resilient so this won't need to be dealt with manually if a duplicate Gerrit CL happens again in the future.

Probably. I think it's a bug that Gerrit was able to get into this state in the first place though. I thought that Gerrit is supposed to ensure a 1:1 correspondence between Branch+Change-Id and CL. I'm not sure how @aclements was able to create those duplicate CLs in the first place.

Ok, I could reproduce it by hitting the maintapi server directly, after some tries:

$ maintq try-work
rpc error: code = 2 desc = grpc: HTTP status 404 Not Found; Multiple changes found for build%7Emaster%7EIb1c7ae1e914116dd8a4440db8ee46d6af3ed1ad5%0A
exit status 1

It succeeds more often than fails, so maybe this isn't what's causing TryBots not to start. (Although it's possible it is what caused coordinator to get into a bad/stuck state, at least in its ability to start/complete TryBot runs.)

I think a good next step will be to restart coordinator and see how behaves, before we try to do more. I'll do that now.

Restarting alone doesn't seem to make a big difference, TryBot runs are still reaching "Builds remaining: 0" and not completing (e.g., https://farmer.golang.org/try?commit=0ca67663 has reached that state some time ago and is still "Active"). This seems to be a problem that needs to be understood+fixed in addition to the "findTryWork" one.

This needs some more investigation, and I'll resume working on it tomorrow morning since it affects TryBot runs only.

Change https://golang.org/cl/279672 mentions this issue: maintner/maintnerd/maintapi: switch to project~numericId Gerrit ID type

I've deployed CL 279672 for the problem spotted in #43312 (comment), and I'm seeing much TryBot activity has resumed.

This problem is resolved now.

I've filed 3 issues for some follow up tasks:

• x/build/cmd/coordinator: add health item for work queue size and errors in finding work #43326—x/build/cmd/coordinator: add health item for work queue size and errors in finding work
• x/build/cmd/coordinator: make logic for cleaning up TryBot runs that reach "0 builds remaining" more robust #43323—x/build/cmd/coordinator: understand why TryBot runs sometimes fail to complete after reaching 0 builds remaining
• x/build/cmd/coordinator: failing to complete TryBot run for change with duplicate Change-Id line #43324—x/build/cmd/coordinator: failing to complete TryBot run for change with duplicate Change-Id line