My theory (sorry, it's late, so not sure if it's obviously right or wrong... also, this bug report is poorly written but I'm pretty frustrated and confused by all of this)

Because we are allowed circular dependencies on modules, we have two libraries:

A@0.1 requires github.com/golang/lint
A@0.1 requires B@v0.0.0-...

B@0.1 requires A@0.1

A fixes its import to golang.org/x/lint, tags 0.2

B@0.2 updates its A dependency, and now requires A@0.2.

But A@0.2 still requires B@0.1 which requires A@0.1 which requires B@v0.0.0-... which requires github.com/golang/lint.

Of course, due to MVS, we only choose one version of the dependency at build time. However, go get -u doesn't seem to use MVS in the same way. It appears to touch every single version in the never-ending chain of dependent versions, even if they aren't used in the actual build.

So the only to break this cycle is to simultaneously tag new versions of A and B, and have them require each other. (incompatible with CI)

In reality, we have more than just two things depending on each other, there's far more. golang.org/x/oauth2, google.golang.org/api/grpc, cloud.google.com/go, google.golang.org/api, go.opencensus.io basically all have dependencies on each other.

/cc @bcmills

We've discussed this general issue today (or yesterday) and Bryan said he might be aware of a fix that can be applied to cmd/go to make the situation better. I'll let him provide more information on that.

Something along the lines of this, maybe? (see patch below)
Tests don't pass, though, so this is probably not sane.

The idea is to not try to upgrade something when we already use a newer version. Theory: upgrading from 0.2 to 0.3 is less work than 0.1 to 0.3, and there's no need to check the same package if you already checked a newer version.

edit: ah, yes, theory and tests fail because vN+1 may have downgraded one of its deps.

@@ -85,6 +89,17 @@ func buildList(target module.Version, reqs Reqs, upgrade func(module.Version) mo
        )
        work.Do(10, func(item interface{}) {
                m := item.(module.Version)
+
+               var skip bool
+               mu.Lock()
+               if v, ok := min[m.Path]; ok && v != "" && reqs.Max(v, m.Version) == v {
+                       skip = true
+               }
+               mu.Unlock()
+               if skip {
+                       return
+               }
+
                required, err := reqs.Required(m)
 
                mu.Lock()

Simplified problem...

M has a requirement on A@0.1 and B@0.1
B@0.1 has a requirement on C@0.1
B@0.2 dropped the requirement on C entirely
A@0.1 has a requirement on B@0.2

"go get -u" still tries to upgrade C, because of M's dependency on B@0.1, even though M won't use B@0.1 or C@0.1 in the build at all, only A@0.1 and B@0.2.

Another ecosystem problem is "indirect" entries in go.mod. As evidenced by zipkin including an indirect entry in github.com/golang/lint:

A@0.1 requires C@0.0
B@0.1 requires A@0.1 and adds an indirect entry to C@0.0
A@0.2 drops the C requirement.

M requires A@0.2 and B@0.1, indirectly depending on C@0.0 even though it isn't used to build, because MVS upgrades B's A@0.1 requirement to A@0.2.

In this case, A = grpc or genproto, B = zipkin, C = github.com/golang/lint

FWIW, I also suspect (a) something is wrong here, or (b) that the behavior here can be improved.

Potentially related is #30455 (comment), which asks if golang.org/x/lint vs. github.com/golang/lint could be viewed as a non-fatal error:

Question: could this be viewed as a non-fatal error? In the original example reported in golang/lint#436, I think there were other require directives for google.golang.org/grpc for versions higher than the problematic google.golang.org/grpc@v1.16.0. In other words, the problematic
google.golang.org/grpc@v1.16.0 I think was not going to end up in the build list if the build list had been allowed to be constructed and the build had been allowed to happen instead of a fatal error.

Also potentially related:

• the comment starting at cmd/go: go get -u in module mode can fail and produce output that's not easy to diagnose #30455 (comment) and the next 10 or so comments.
• cmd/go: get -u updates more in module mode than GOPATH mode #26902 "cmd/go: clarify which dependencies are updated by 'get -u' without '-m'"
  • including cmd/go: get -u updates more in module mode than GOPATH mode #26902 (comment)
• cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702 "cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u'"
  • including cmd/go: 'mod tidy' downgrades an unused indirect dependency upgraded by 'get -u' #29702 (comment)
• cmd/go: building requires otherwise-unused .mod files to be present  #29410 "cmd/go: building requires otherwise-unused .mod files to be present"

Leaving the "too much upgrading" problem for #26902.
Retitled this to be about lint since that is the specific complaint in "what did you expect to see?".

It seems like we need a clear signal from the upgraded module that "I used to be this old name". And then if you are resolving the old name and land in the new module claiming "I used to be this old name", the build would automatically interpret the old name as the new name. Maybe that's a per-module signal; maybe it's a per-package signal. It needs to be an explicit signal, though. It's not enough to just "find code at X that seems to be module Y so pretend X means Y" because that breaks people who github fork X into their own account as Y. In that case we don't want go get of the fork to silently redirect back to the original.

It seems like we need a clear signal from the upgraded module that "I used to be this old name".

This requires the upgraded module to know its old name. In this specific case, it might be possible, because the canonical import path of lint used to be github.com/golang/lint a long time ago (see here.

However, it didn't start enforcing import paths until much later (via import comment for GOPATH mode, and via go.mod file for module mode), so some projects may have started importing it via any other alias, not just github.com/golang/lint.

Should it be responsibility of an upgraded module to know what were all the non-canonical import paths that some projects in the wild may have imported it as?

Should it be responsibility of an upgraded module to know what were all the non-canonical import paths that some projects in the wild may have imported it as?

In general that set is well-defined: we must have arrived at the repository somehow (via redirects), so the only possible names for it are things that redirect to it. I would guess that it's pretty rare for folks to set up redirectors for arbitrary repositories just to use them via different import paths.

One short description of the build list (from the cmd/go documentation) is:

The build list initially contains only the main module. Then the go command adds to the list the exact module versions required by modules already on the list, recursively, until there is nothing left to add to the list. If multiple versions of a particular module are added to the list, then at the end only the latest version (according to semantic version ordering) is kept for use in the build.

It is unlikely that I will describe this exactly correctly, but from the outside it seems that if the inconsistency between golang.org/x/lint vs. github.com/golang/lint could be viewed as a non-fatal error, it could mean that the particular problematic version of the consumer does not get added to the candidate build list being iteratively built up. In other words, don't add the the version of the consumer with the import of github.com/golang/lint because that version of the consumer has an error, but that error is not fatal to the overall build.

Or maybe that is not the right way to think about it -- maybe instead the "bad" consumer version gets added to the build list but is marked that it is bad, but that is not a fatal problem for the overall build as long as the bad version is not the version that gets selected in the end.

It seems some type of approach along those line would have solved the original x/lint issue reported in golang/lint#436 (given google.golang.org/grpc@v1.16.0 was the "bad" version that imported the wrong flavor of lint, but there were other require statements for v1.17.0, v1.18.0 for grpc in the overall build and hence the v1.16.0 of grpc was not going to be selected anyway). Perhaps that would also help the more recent incarnation being reported here by @broady.

Edit: part of the problem with the current behavior is the degree to which old problems that were fixed a while ago are still a "hangover" for right now. E.g., grpc I think fixed the lint problem in its own go.mod over 4 months ago and released multiple releases since, and it seems a healthy portion of the ecosystem has started using those newer releases of grpc... but one laggard grpc consumer seems to be fatal right now (which seems to be in contrast to how usually in a modules world "laggards" are gently pulled forward by the rest of the ecosystem over time).

It seems like we need a clear signal from the upgraded module that "I used to be this old name".

I'm not sure this solves the problem. Suppose there's some requirement on some version of a module that points to the bad pseudoversion of lint that doesn't include the old name. cmd/go still tries to validate that bad version and fails.

I think it needs to be treated as a non-fatal error (non-fatal because it doesn't end up in the final build graph).

exclude works fairly well here as a workaround, could this bad version of grpc/lint get treated in the same way as excluded module versions?

Suppose there's some requirement on some version of a module that points to the bad pseudoversion of lint that doesn't include the old name.

The problem is “does include the new name”, not “doesn't include the old name”. The problem today is that folks depend on revisions of github.com/golang/lint from before the go.mod file was added: those are allowed, but cannot be upgraded because the latest revision has an explicitly different path.

If the latest revision instead has an explicit alias declaration, then everything would work fine.

The problem is “does include the new name”, not “doesn't include the old name”.

Understood. I was illustrating a situation* where proposed fix of "module can specify acceptable old names and/or aliases" might not work. I suppose that's an edge case, since it would mean an invalid go.mod file in the first place (e.g., if someone had manually changed golang.org/x/lint to github.com/golang/lint).

'* the case where something depends on a version of github.com/golang/lint that doesn't specify its aliases.

exclude works fairly well here as a workaround, could this bad version of grpc/lint get treated in the same way as excluded module versions?

I don't know if exclude is the right analogy, but following that thought for a moment -- rather than lint (in any form) being treated as if it was excluded... would it be possible to treat grpc@v1.16.0 as more or less excluded (by virtue of the fact that grpc@v1.16.0 references a old spelling of lint that fails to upgrade because of the mismatch with latest)?

The problem today is that folks depend on revisions of github.com/golang/lint from before the go.mod file was added: those are allowed, but cannot be upgraded because the latest revision has an explicitly different path.

Perhaps a completely different approach would be -- rather than saying the old spelling of lint cannot be upgraded, could the old spelling of lint be allowed to upgrade to the version of lint just before the go.mod file was added to lint? In other words, latest version of lint has a disallowed inconsistency with the old import path, but a prior commit does not, so emit a warning but do the upgrade to the valid prior commit of lint (which might be a no-op). If that was the behavior, in the scenarios here I think that would translate to the build completing successfully (and given grpc gets pulled up to more recent versions of grpc due to other require directives, the old spelling of lint would be dropped from the final build list here, I think?).

could the old spelling of lint be allowed to upgrade to the version of lint just before the go.mod file was added to lint?

Absent explicit release tags, we would have to scan every commit to the repository to find such a version.

What is the solution?

@DukeAnn

What is the solution?

As far as I am following, I think this issue here is more about a medium or long term fix by changing the go tool, but that is not happening immediately. However, there is a healthy chance there is a more immediate solution to whatever you are seeing.

It probably would make sense for you to review the symptoms at googleapis/google-cloud-go#1359 and golang/lint#436 and see if your symptoms match one of those reports, and either add comments there or perhaps open a new issue.

This has surfaced again in googleapis/google-api-go-client#347 (git.apache.org/thrift.git became github.com/apache/thrift).

AFAICT, without the Go tool changing in some way, library owners have to resort to alien things like making libraries rely on non-existent versions as kindly spelled out by @dmitshur here or temporarily omitting dependencies. Both of these solutions require a lot of coordination and releasing multiple tags across multiple.

This is pretty painful. :|

Are there any other medium-term solutions anyone else knows of?

Change https://golang.org/cl/220080 mentions this issue: design/36460: add design for lazy module loading

We've given this issue a lot of thought over the last several years. The current best-practice solution for changing a module's import path is to choose at least one of:

• Give the new path a different major version from the old one, so that the tags for the two modules are in separate spaces.
• Or, clone the old repo to the new path instead of renaming it, leaving the tags and branches of the old repo at an appropriate point for the old module path.

Optionally, also leave behind packages in a repo at the old path containing declarations that merely alias (or forward function calls) to the packages at the new path.