cc @heschi

I wonder if there is some sort of retry logic that should be applied here in the case of connection refused?

#28194 is probably relevant for that, but it's tricky: I would expect that an import-path server generally fails due to (say) temporary outages for machine or infrastructure updates more often than nondeterministic errors or short-term network drops. How do we balance retrying flakes vs. stalling on retries that won't succeed any time soon anyway (compare #40376)?

What's more, transparent retries can hide real problems — masking problems generally provides a short-term benefit at a long-term cost. (Compare #42699, and “Postel was wrong” is a good read too.)

#42809 might also be relevant. I can understand it if we don't want to keep retrying with short intervals.

That said, I think we can do better without wasting resources. For instance, like Paul said, one of the servers saw the version while another didn't. This resulted in inconsistent yet reproducible behavior among users, which we could reproduce for a solid ten minutes. I think that kind of inconsistency could lead to significant confusion in practice; imagine an user reports to upstream that go install always fails for them, but for upstream it always works.

At a conceptual level, I imagine that as soon as one of the servers sees a valid new version, it could invalidate cached errors from all other servers.

There is only one instance of the mirror, but it is a globally distributed system with multiple layers of caching. We're not trying to guarantee global consistency, so this is working as expected as long as the cache became consistent within half an hour. There is no way to synchronize the global cache.

@bcmills thanks, #28194 and #40376 are indeed relevant. But I think somewhat different. Because in this case the caller is not in control of the side effect of the (30 mins, albeit temporary) "bad" cache state that might result, yet they will likely suffer its (temporary) "long"-term cost, as will others. So for this specific error, one could argue that retrying and the stalling that results in is worth it.

@heschi thanks for the detail.

Is it always the case that the {proxy,sum} mirror reflects the first fetch of a module, or rather one of the fetches during the 30 minute window that follow the first fetch?

Regardless of whether cmd/go is changed in some way to retry, in the case of this specific error is there merit in having a shorter cache invalidation period (apologies if I'm not using the correct terminology)? Again the reasoning here being that the caller, all else being equal, is very likely not in control of this category of error.

The only guarantee we attempt to make is that users see data that's less than half an hour old. There is no concept of a "first" fetch in the system. So if it does multiple fetches within a 30 minute interval, users may see any of the results in the interim.

The go command doesn't provide any kind of structured output for fetch errors, and the mirror has so far avoided parsing error text. I don't think this is a critical enough case to start doing that.

There is no concept of a "first" fetch in the system. So if it does multiple fetches within a 30 minute interval, users may see any of the results in the interim.

Obviously not to question your explanation, but rather to share my understanding to this point which has been that the first fetch was the significant one. However it sounds like there are some scenarios where {proxy,sum}.golang.org can settle on a later module version that is different from the first fetch version, i.e. following a tag move?

The go command doesn't provide any kind of structured output for fetch errors, and the mirror has so far avoided parsing error text. I don't think this is a critical enough case to start doing that.

I'll defer on this point, because they only data points I am aware of are my own issue report.

Not all fetches are successful, as in this issue. And not all names stay the same forever, e.g. branches. But we are now way off topic for this issue.