There are 2 gitmirror instances. This was resolved by restarting the failing gitmirror instance (deleting the pod), which we discovered by proxying to the pods and investigating their statuses.

One idea would be to add stackdriver metrics to gitmirror for monitoring this. Another would be to improve the health check so that it checks all the pods, rather than relying on the kubedns load balancing to land on a healthy one.

This issue is happening again now. Refreshing farmer.golang.org sometimes shows that Git mirroring is okay, other times it's showing "repo go: hung? no activity since last success 8h41m49.610989692s ago":

There isn't a real mirroring problem, because a 4-hour commit in the main Go repo is successfully mirrored to GitHub.

From looking at the logs for the 2 gitmirror instances, I see a number of errors from the Gerrit servers today. That may be what caused at least one of the instances to get into a bad state.

Change https://golang.org/cl/226678 mentions this issue: cmd/coordinator: report errors on all gitmirror instances

to improve the health check so that it checks all the pods

I've sent CL 226678 that implements this.

Tested the CL above, it worked as expected:

No more flakiness, refreshing the page always produces the same result. 🎉

I've redeployed gitmirror to resolve its current issue. Now that the reporting at https://farmer.golang.org/#health is reliable, we can deal with gitmirror problems if they come up in the future.