I'm still not convinced that long-term use of replace — other than by developers during development of their own modules — is actually something we should encourage.

(Note that we do support it, in the sense that it remains possible to check out the repository and run go build within its working tree, but dedicating a go subcommand or flag to it would significantly change the perception around when and how replace directives should be used.)

In particular, the compatibility argument for using replace seems off too me: if someone is building your module as the only thing passed to go get, then they get exactly the module versions that you've selected — so even if there are breaking changes after that point, why are they relevant?

So I would like some more detail: @rogpeppe, why does juju require replace directives today, and what is preventing you from migrating away from them over the long term?

So I would like some more detail: @rogpeppe, why does juju require replace directives today, and what is preventing you from migrating away from them over the long term?

OK, here's some detail on the replacements used by Juju currently. I speak at some remove, because I haven't been directly involved in the Juju code base for a little while now, but I have been historically familiar with some of the issues.

Summary

It might be possible to move away from replace statements in the long term, but it's not necessarily possible in practice without hard-forking multiple external repositories, which has its own on-going cost. It's certainly not going to happen in the short to medium term, which means that replace statements are going to be a necessity for building juju for a while to come.

And even the need for these particular replace statements is dropped eventually, another similar issue might well arise in the meantime. The open source world is not always clean. Some kind of replace statement is a crucial band-aid at times, and one which anyone installing the juju commands, not just developers, will need to apply.

github.com/altoros/gosigma => github.com/juju/gosigma

Fixes races in tests and code. https://github.com/Altoros/gosigma/pull/1/files.
No new or changed API.

There's been a PR open since 2017-05-23 without response.

Making a hard fork of this probably wouldn't be that hard to do, although might
be politically awkward, as github.om/altoros/gosigma is the "official" API client.

gopkg.in/yaml.v2 => github.com/juju/yaml

A tweak to the accepted floating point syntax. No new API. A small but
significant tweak - the two repositories disagree on whether the a
member in:

a: 123456e1

holds a floating point number or a string. Changing this behaviour might
break backward compatibility with existing YAML files that are held in
immutable databases online.

Changing this unilaterally inside juju could break other code. For
example, code might return a yaml.MapSlice which is intended to be
marshaled as an order-preserving map, but would instead be marshaled as
an array because its type is not recognized.

Changing this across the entire code base would be hard, as many external
repositories have dependencies on this package.

gopkg.in/mgo.v2 => github.com/juju/mgo

19 commits. Some new API.

The parent repository is no longer maintained, and was not generally
accepting of patches even when it was. The main fixes here are to the
mgo/txn code, which is heavily used by Juju and has been the source of
some major operational issues. The patches address some of these issues,
amongst other things.

Changing this dependency to use github.com/juju/mgo throughout
would be a major piece of work. It is a dependency of many external
repositories which don't themselves care about the fixes that Juju
needs (because they don't use mgo/txn); changing the import path in
those repositories would require a major version change, and subsequent
cascading changes. Persuading the owners of all these repositories that
they should switch to Juju's fork of mgo for fixes that don't directly
affect them would require significant politics, if it could be achieved
at all.

Hi @bcmills, briefly cherry picking a couple points you made:

In particular, the compatibility argument for using replace seems off too me: if someone is building your module as the only thing passed to go get, then they get exactly the module versions that you've selected

...unless of course part of the way you selected the module versions as the binary author is via a replace or exclude directive, in which case your consumers don't get the module versions that you've selected if your consumers do a go get under the current system.

so even if there are breaking changes after that point, why are they relevant?

That is not my concern -- modules provide 100% reproducible builds, etc.

The compatibility argument is not about future breaking changes causing problems for consumers, and is more about incompatibilities that exist when a binary is being published given for example (a) semver is not always perfectly followed in practice and (b) people use v0 dependencies from the "compatibility free zone". Either of those mean MVS by itself can select problematic versions of foo's dependencies that the human author of foo can resolve with replace prior to publishing foo. As Russ wrote, "A real system needs more flexibility, for example the ability to exclude certain module versions or replace others". It seems to me incompatibilities are things that arise in a real system in ways that MVS (by design) cannot always resolve by itself.

unless of course part of the way you selected the module versions as the binary author is via a replace or exclude directive

That's a bit of a circular argument, though. 🙂

The central question here, as I see it, is: should replace directives be a tool for selecting module versions, or only for relocating them? (I can see a clear use-case for relocation: namely, local development of dependencies. The use-case for selection seems less clear-cut, because it partially overlaps with the usual version selection by MVS.)

@rogpeppe, thanks for the detail.

It seems to me that since gosigma and mgo are effectively unmaintained, they should probably be hard-forked (if they cannot be transferred to more active maintainers). That makes replace (and especially #26904) an important migration tool, but a temporary one.

Specifically in relation to mgo, you note:

changing the import path in those repositories would require a major version change, and subsequent cascading changes.

Could you give some more detail about why changing the import path would require a major version change? (Do packages pass through mgo types in their exported APIs in a way that cannot be transitioned using type aliases, interfaces, or a mix of the two?)

The issue with yaml seems very unfortunate, but if the two packages fundamentally disagree on the meaning of certain YAML files, then it seems even more important that dependencies outside your project receive the version of the package they were developed against, in order to consume their own YAML files (if any) correctly.

That is, I suspect you'll want to maintain a fork of the package for compatibility with existing YAML files, but you'll want to use that fork only for the parts of the code that consume existing files.

On the other hand, if you replacement really is benign for your dependencies, then perhaps the yaml package should have an explicit, supported hook (a runtime function call, or perhaps a build tag) that you could use to configure the mainline package. (Have you discussed that possibility with the go-yaml maintainers, presumably @niemeyer?)

Could you give some more detail about why changing the import path would require a major version change? (Do packages pass through mgo types in their exported APIs in a way that cannot be transitioned using type aliases, interfaces, or a mix of the two?)

Yes. For example see this package. The forked mgo package can't type-alias the original Collection type (because then it can't define its own methods on it); neither can the mgorootkeystore package change the type it's using to an interface.

Changing the mgorootkeystore package to use a different version of mgo would definitely be an API-breaking change.

I agree that replacements are not desirable and can usually be worked around in time (with negotiation with upstream maintainers, hard forks etc), but the fact remains that they are important at the time for making working products.

Even if the replacement is just a "temporary" fix, it can be an important one, and I consider it important that it's possible to publish a predictably-versioned Go binary with the fix in place until the replacement can be removed. Telling the user to do their own git clone (working out the correct incantation for that is non-obvious with vanity import paths) or similar is not a good solution.

I think Russ's words, aptly quoted above, bear repeating once again:

Minimal version selection is very simple. It achieves simplicity by eliminating all flexibility about what the answer must be: the build list is exactly the versions specified in the requirements. A real system needs more flexibility, for example the ability to exclude certain module versions or replace others.

Please, let's provide that flexibility for all users of Go modules.

BTW I would be perfectly happy if file-path replacements yielded an error in this case. I do consider file-path replacements suitable for developers only (and they're not a fundamental part of VCS, which replacements and exclusions in general are).

This has an interesting interaction with #33848.

#33848 proposes to use the main module's vendor directory automatically when it exists, in much the same way that we use the main module's replace and exclude directives. However, the main module's vendor directory — much like filesystem targets of replace directives — may include go.mod files that cause it to be pruned out from the module's .zip file.

Also like a replace directive, the vendor directory may include local edits and patches, although I would not recommend patching the vendor directory in that way.

Perhaps #26904 provides a better way forward: one option we're considering there (from among many alternatives) is adding some new keyword to the go.mod file, which would remap or alias the import paths themselves rather than their source code.

Those paths are necessarily always import paths, not filesystem paths, which obviates the concern about treating filesystem paths and module paths differently.

Unfortunately, applying only a subset of the main module's directives still poses a testing problem: it would still mean that the module could be built — and would need to be tested in — a third configuration that is neither the same as the view from inside that module, nor from an external module.

The more I think about it, though, the more convinced I am that an “install from semi-inside” command should not be go get with a flag.

go get currently serves two purposes: it resolves and adds dependencies, and it installs binaries from within those dependencies. A command to install binaries from within one specific module is meaningfully different: it constrains the module relationship of the arguments in a way that go get in general does not.

You might even imagine that a build in this mode should not resolve missing imports: if the point is to exactly reproduce what the developer would have built, then the developer necessarily needs to record all of the versions that they intended to use.

You might even imagine that a build in this mode should not resolve missing imports: if the point is to exactly reproduce what the developer would have built, then the developer necessarily needs to record all of the versions that they intended to use.

This seems like a good idea to me.