/cc @bcmills @myitcv

Why not fix this with tooling? For example, have you tried @rogpeppe's https://github.com/rogpeppe/gohack?

What is the underlying problem you want to solve?

Customization hooks add complexity, to both the implementation and the user-facing documentation, and the documentation for modules is already much more complex than I'd like. We should only add hooks if they provide a substantial enough benefit to offset that complexity, so we need to understand the concrete benefit you're hoping to provide.

@bcmills The underlying idea of adding using directive is to allow end configuration to go.mod. Especially when compiling for different conditions (platforms). The problem of the current replace directive is that it generally introduced a hard-coded path which is not portable. But the path is actually variable and unpredictable.

For example of this:

module example
require example.io/foo v0
replace example.io/foo => /path/to/my/module/base

It works for one dev, but complains for the other having /my/module/base instead of /path/to/my/module/base.

The other benefits of having using directive to introduce the variable is that users can mimic the GOPATH approach as like setting the replacement target as wished by operating devs. For example:

$ go build

and

$ MYDIR=/my/module/base go build

should work perfectly for this module differently:

module example

using env MYDIR /default/module/base

require example.io/foo v0.0.0
replace example.io/foo => $MYDIR/foo

I'm not sure if I understand the customization hooks you mentioned. But I think such using directives and $VAR should be just expended to a plain go.mod text (no $VARs) in memory and then being further processed by what it was like.

This brings great flexibility when using go mod and the go-module enabled go tools (go build, go install, etc.). The using directive and $MYDIR variables should not affect what go-modules have already done for go tools. It just improved the way that go.mod was interpreted.

Such an improvement will allow authors to define their own project-specific building variables (MYDIR) and allow taking variables from other build systems without the needs of touching go.mod (to fix the replacement target). Which is very useful and important.

Introducing the using variables should also help improving documenting the build process. It should be much easier to document the using variables then telling how to edit go.mod to the right form before we can correctly get go build done.

I would consider creating a "template" file such as go.mod.in that would have the placeholders such as replace example.io/foo => MYDIR/foo and then using a sed command or Perl/Python/Go script to replace the placeholder with the desired value and generate go.mod. Personally I would also be against adding more complexity and customisation features to the modules file as it is increases cognitive load on users (and implementation complexity); also in this case it would make go.mod file not self-contained but dependent on the environment settings.

@dmitris @bcmills I think when we're building a go program. We generally do go build directly (manually typing from a terminal) or indirectly (called from an other build system or scripts).

When we're building directly, aka. typing go build from a terminal, we will want to avoid hard coding the replacement target. For example of my presentation like example.io/foo => $MYDIR/foo. If all developers are from the same small office, it should be very much easy to comply to a common workspace path, e.g. /my/team/project. In this case the go.mod could work without touching any single line by defaulting MYDIR like this using env MYDIR /my/team/project. But when we're working with external hands, it might be very difficult to force others to go with /my/team/project setting. The go.mod of version go1.11 will ask us to fix replacement target first manually. I assume the maintainer will write some lines of docs telling about fixing the replacement target. Or in advance, (s)he might just write a helper script to do an automatic fixing and ask audiences to use that helper script, e.g. build.sh or makefile instead of calling go build directly. In this case, the scripts might be looking like this:

#!/bin/bash
# helper build script `build.sh`
sed 's|MYDIR|/a/variable/location/calculated/somehow|' > go.mod && go build

# helper `makefile` working with `make`
build:
        # do something to fix the `go.mod` file
        go build

It will definitely work and shouldn't be very difficult to apply. Or the author could also ask the audience to fix the replace example.io/foo => $MYDIR/foo line using their favorite editor manually.

When we're building indirectly, a hardcoded replacement should be easily fixed by other build systems, for example of seding the replacement target, but still the build is not portable. When introducing the using env/arg directives to make replacement targets variables/configurable, it should give go build a common interface to replacement the extra steps of seding replacement targets.

Another benefit when we have using env/arg is that we could mimic/simulate the GOPATH approach for each project. Like in the past, all we need to do is setting GOPATH to the right location and all go build works that easy way (but it has already been broken by things like .godep, which is really annoying as it's requiring the external godep and extra steps to make it happy). The problem of old GOPATH is about versioning the dependencies (vendoring). With using env/arg directives, we're possible to do similar things like old days, the difference is that we use MYDIR rather than GOPATH, and different projects could have their own MYDIRs.

# .bash_profile
GOPATH=/my/favorite/go/path
MYDIR=/my/favorite/project/path
MYFOO=/my/favorite/project/third_party/foo

So that the using env/arg for go modules respects our go habit (the go-fashion) and the versioning issue:

$ go build -o were_all_happy

This is very important when our required module is too deep in the require graph. It's really frustrating to grep out the location and fix the right go.mod to make go build happy. And yes, it's possible to make use of a template technic, but are you going to do templates every time we introduced a third party module? Or better if that third party module had defined a MYDIR for it's users? So that we can do:

$ MYFOO=/dont/complain/please go build -o all_right

or

$ go build -mod-arg=MYFOO=/dont/complain/please -o all_right

Questions

• But are these extra steps really necessary when we're working with go?
• What if it goes recursively? Say if a required module needs a hardcode fixing too before go build is happy.
• Is the using env/arg directives making things over complex or brining authors more flexibilities?
• What using env/arg really changed to the go tools?
• What else should we complain?
• The replace directive is very useful for non-hosted or private repositories. Is that possible to ask/suggest authors not to put a replace directive at all? This should require all modules being hosted (github.com or example.io, is it easy to setup a custom example.io server?).

@duzy, I still don't understand what concrete problem you want to address with these replace directives. Why do all of the developers in your office need to replace some module with the same path?

• Is it so that they can make edits to two modules simultaneously? (If so, that's cmd/go: support simultaneous edits of interdependent modules #27542, and in the interim you might want to see if github.com/rogpeppe/gohack addresses your needs.)

• Is it because you require a customized fork of some upstream dependency? If so, it seems like the replace directive ought to point to some stable module path (possibly hosted on a local GOPROXY server) rather than a local directory.

• It it some other reason? If so, what?

@bcmills The replace is useful for us in different cases while using go.mod. We're generally working on multiple projects. Here're some example cases of using replace directive:

• We're working with some common modules, which is in used by different projects. I think if we published these modules to GitHub (or other hosts), it should be working fine. But we're using the modules in-house only. So that replace came into help, supposed that example.io is our own repos site, we solve the issue like this:

module foo
require example.io/bar
replace example.io/bar /my/local/source/bar

This works perfectly me, but not for all in-house devs. Because devs could place bar in anywhere they want. So that all devs will have to change /my/local/source/bar to their favorite, and it conflicts every time they push the modification. In order to fix such conflicts, I proposed the using directive to rescue our life:

module foo
using env MYPAT /my/local/source
require example.io/bar
replace example.io/bar $MYPAT/bar

If this works in go1.12, you will save our life in this case.

• An other example of using directive is that when we made some edits to a third-party module, say github.com/example/bar, we have to replace it somewhere else, say example.io/third/bar, to maintain our revisions:

module foo
using env MYPAT /my/local/source
require example.io/third/bar
replace example.io/third/bar $MYPAT/bar

I believe there could have many cases of using replace. The point of using go.mod instead of third-party similar tools is that we want to make life easier. It's our preference to using go tool rather then introducing more third-party tools.

I wish this help you to find out why the proposal useful.

@bcmills By the way, I have another proposal which is more interesting. But I'm not sure if you like to add it. The second proposal is inspired by go test, which reads test cases from files like foo_test.go, so why not we do the similar approach rather then using go.mod files.

For example, say if we have the package go/mod or go/tool/build (similar to testing), and we write go.mod information in special Go files, say go_build.go or go_mod.go, and we define go.mod information in Go code instead of introducing go.mod directives:

package modinfo // or any special name preferred
import (
    "go/mod"
)
func init() {
    var s = "example.io/third/bar"
    // do any calculation for `s`...
    mod.Require(s)
}

Or

package modinfo // or any special name preferred
import (
    "go/tooling"
)
func GoRequire(m *tooling.Mod) {
    var s = "example.io/third/bar"
    // do any calculation for `s`...
    m.Require(s)
}
func GoBuild(b *tooling.Build) {
    b.Run()
}

In Go code like this, we're possible to do anything we want! All we need to do is to populate go mod in Go API go/tool/build or go/mod, and using the similar approach as go test calls Test*:

$ go build # it calls GoRequire and GoBuild

This approach could allow us to complete very much more interesting jobs related to building.

@duzy - if this is all in-house, why not put everything in a single repo? Then this problem goes away entirely, as does the overhead of maintaining separate repos. Especially if you are checking in go.mod files with such replacements, i.e. there is something permanent about them.

If these replace directives are temporary, then I think what you're effectively asking for is covered by #26640. With #26640 you would (via gohack or some similar approach) be able to do the environment variable expansion you want.

We're working with some common modules, which is in [use] by different projects.

In general the expectation is that those be published somewhere, even if it's only accessible within your local network. (For example, if your company is example.io, you could have the server for example.io detect requests from internal IPs and serve meta tags pointing to an internal version-control server.)

If you need to edit those modules simultaneously, well-known environment variables don't seem like a scalable solution: if you need to fix an upstream module, how would you know which variables you need to set to work on it? I'd rather we address that use-case directly; see #27542.

when we made some edits to a third-party module, say github.com/example/bar, we have to replace it somewhere else, say example.io/third/bar, to maintain our revisions

That should be fine today: the target of a replace directive can be another module path. (It doesn't have to be a filesystem path.)

say […] we define go.mod information in Go code instead of introducing go.mod directives

I believe you can already do that today using a Git post-checkout hook or similar.

@duzy - if this is all in-house, why not put everything in a single repo? Then this problem goes away entirely, as does the overhead of maintaining separate repos. Especially if you are checking in go.mod files with such replacements, i.e. there is something permanent about them.

@myitcv I think in-house (office) development and separating the system into multiple repos (packages) are two different kind of problems. To decide whether you should go all in the same repos or split the system into many, it's up to the scale of the project or the design principle of the system. To me, if a part of the system is reusable to others or useful (feature) in the future, I would consider move it into a new package. It could make thing easier rather then introducing maintaining costs. In-house is not a good suggestion that everything should be in one repos.

The changes to those replace directives should not be called "temporary", it in fact a crucial step in the flow because without replace go mod don't work properly at all.

I think yes, if you're not going to add my second proposal, I'm interested to make it possible through the gohack approach. I think defining the go mod meta information or logic in go_build.go (or go_tooling.go) files is more interesting then using go.mod files. It works just like go test. It's the go flavor.

We're working with some common modules, which is in [use] by different projects.

In general the expectation is that those be published somewhere, even if it's only accessible within your local network. (For example, if your company is example.io, you could have the server for example.io detect requests from internal IPs and serve meta tags pointing to an internal version-control server.)

@bcmills Ideally, yes, by having our own example.io to serve as hosted-replacement is a practical approach and meets the package naming scheme [host]/[path]. But not all of us like it, reasons:

• it requires to have real domain name and hosted meta tags plus potential bandwidth for transferring packages. I once did godev.io for my own purposes and it's working the same way as gopkg.io does but more flexible extensions. But turned our I'm not using godev.io again, because it costs me money on bandwidth to delegate the packages from another host like GitHub.
• not always do we need to configure a remote setting to import a package -- I think this is one of the reason why replace was implemented.

I believe you can already do that today using a Git post-checkout hook or similar.

I think git hook could be a solution too. But I prefer to see if my second proposal is possible now as it's more interesting than go mod. If go build can hook into my own go code GoRequire or GoBuild to complete a custom modification or logic, it could replace gopkg.in, godev.io and go mod in go flavor. So I'm interested to look into this when available.

So could you please give me some suggestions on how to implement this idea:

package foo_tooling
import (
    // or "cmd/go/tooling" to make use of some features in "cmd/go/internal"
    // without an official tooling package, we might use the "gohack" approach or just
    // move custom implementation into "cmd/go" to make use of the cmd/go/internal
    "go/tooling"
)
// helps build, get, download, etc.
func GoRequire(a *tooling.Action) {
    // a.Name == "build", "get", "download"...

    var s = "example.io/third/bar"
    var ver = "v0.0.0-20181818000000-xxxxxxxx"
    // do any calculation for `s`...
    a.Require(s, ver)
}
// go build
func GoBuild(b *tooling.Build) {
    // do something meaningful here...
    b.Run()
}

Which work with go build (inspired by go test) that it invoke GoRequire and GoBuild at some point in the build flow?

@duzy

To me, if a part of the system is reusable to others or useful (feature) in the future, I would consider move it into a new package.

Just so we're not talking past each other on a couple of points.

In a pre-modules world, the repository was the unit of versioning. Repositories contain multiple (related) packages.

In the world of Go modules, a module is a collection of related packages. The module is the unit of versioning. A repo can contain multiple modules.

So my suggestion about moving towards a mono-repo is really motivated by the fact that Go modules enable multi-module repos, and it sounds like you have/need multiple modules. Of course you don't need to have multiple modules to justify a mono repo.

The changes to those replace directives should not be called "temporary", it in fact a crucial step in the flow because without replace go mod don't work properly at all.

Please can you share a few more details on your workflow? Because I can't see a reason for replace directives to be anything other than temporary.

If they are more permanent it suggests to me you aren't relying on the version requirement relationship between modules, and instead using replace directives to effectively achieve something akin to GOPATH. Assuming this line of reasoning (and apologies, I'm anticipating your response to my question above), then I wonder why you don't just have a single module and all of those packages in that single module.

I think yes, if you're not going to add my second proposal, I'm interested to make it possible through the gohack approach.

If you do need multiple modules, then yes, I think the gohack approach (with or without go.mod.local) sounds like it will suit your needs. But a few more details will help.

@duzy

To me, if a part of the system is reusable to others or useful (feature) in the future, I would consider move it into a new package.

Just so we're not talking past each other on a couple of points.

In a pre-modules world, the repository was the unit of versioning. Repositories contain multiple (related) packages.

In the world of Go modules, a module is a collection of related packages. The module is the unit of versioning. A repo can contain multiple modules.

I think we have the common understanding on these (modules, repos, packages). Just in the context of my case, I placed my common reusable packages in a place like the GOPATH directory. And I paced the package (in the context of saying) in such a directory (could be in a single repo). They're not belonging to any Go module. I like a place called MYDIR (in the previous posts) for me to contain all extracted useful Go code (in packages). This MYDIR could be in a (managing) repo or being a repo itself. I edit the Go code in MYDIR actively.

So my suggestion about moving towards a mono-repo is really motivated by the fact that Go modules enable multi-module repos, and it sounds like you have/need multiple modules. Of course you don't need to have multiple modules to justify a mono repo.

I think your suggestion is very positive and useful actually and the motivation is rational to me. I really appreciate that and I'm very happy to talk. I think in my context things are different as I'm not only working with Go modules and go-module is different then repo in my environment. So I think using the term go-module should help identifying the problem of replace with hardcoded paths (but not confusing with repo as a go-mod dir and a repo dir could be not related).

The changes to those replace directives should not be called "temporary", it in fact a crucial step in the flow because without replace go mod don't work properly at all.

Please can you share a few more details on your workflow? Because I can't see a reason for replace directives to be anything other than temporary.

So as said above, I have commonly shared packages in MYDIR. These packages are reused by many Go modules (other packages or utilities). In pre-modules environment, I have two options for other packages to import. 1) put those packages in GOPATH or 2) set GOPATH to $MYDIR:$GOPATH.

In Go module environment, things are different as it's not searching into GOPATH for modules. As we all know that Go module use a different approach to manage mods, and all downloaded mods are non-editable.

So my pre-module workflow was broken. Luckily the replace directive came into my search. I have to turn all packages into Go module (by initializing go.mod files). One additional benefit of using replace is that I could use a 'fake' module root/host. For example, I like to use a example.io/foo/blah import scheme while coding in Go. I don't need to care if example.io really exists or not. Because I'm making it working all right with replace directives. For example:

package main
import "example.io/foo/blah" // without Go module and `replace`, it was like "my/foo/blah"
func main() {
    blah.Hey()
}

And in go.mod I'm doing this:

module foo
require example.io/foo v0.0.0
replace example.io/foo /local/path/to/my/foo

This works for me to replace the GOPATH approach.

But the problems came into my mind, why do I hardcode the replace destination path /local/path/to/my/foo? This is not a perfect method in my sense. I understand that go.mod was designed to be edited by both human and go tool. But the problem is that only human knows what to replace.

I found that I'm not the only being troubled at this point. And I learned that someone is using go.mod and go.mod.local, go.mod.xxx variants to solve. But my question is still that why it's hardcoded and not a variable.

So that using env and using arg were born in my head. And go.mod should become:

module foo
using env MYDIR /local/path/to
require example.io/foo v0.0.0
replace example.io/foo $MYDIR/my/foo

This will solve things well:

$ MYDIR=/local/alt/path go build -o foo .

I think yes, if you're not going to add my second proposal, I'm interested to make it possible through the gohack approach.

If you do need multiple modules, then yes, I think the gohack approach (with or without go.mod.local) sounds like it will suit your needs. But a few more details will help.

The reason why I'd like to look in hack into "cmd/go/internal" is that I wish to make something like go_tooling.go to replace go.mod. The implementation of Go module (and other go tools) are internal. So that I have to hack into cmd/go to reuse the implementation.

The purpose of using go_tooling.go to replace go.mod is that Go code could perfectly describe what's required from a package/module, so that go.mod directives could be avoided. For example (more examples could be found in the previous posts):

package foo_tooling
import "go/tooling" // or "cmd/go/tooling"
func GoRequire(a *tooling.Invocation) {
    ...
}
func GoBuild(a *tooling.Invocation) {
   ...
}
func GoDownload(a *tooling.Invocation) {
   ...
}

And the package dir could be like:

$ ls
foo.go foo_test.go foo_tooling.go

The go.mod file is replaced by foo_tooling.go.

While using Go code to describe the Go module requirements (require), we're able to have unlimited possibilities.

In this tooling proposal, "_test.go" files are working with go test and "_tooling.go" files are working with go build, go get, go download, etc.

The remaining question is that how "go/tooling" or "cmd/go/tooling" should be designed?

• it requires to have real domain name and hosted meta tags plus potential bandwidth for transferring packages.

Last I checked a real domain name is $12/yr., and I'm not sure that you even need that: a reserved local IP address (even a loopback address to your local machine!) would suffice, as long as the <meta> tags point to someplace that you can resolve.

• not always do we need to configure a remote setting to import a package -- I think this is one of the reason why replace was implemented.

I'm not sure what you mean. If you're importing a package from the same repository, you can use relative replace directives within the module. If you're importing from a different repository, you need to be able to resolve that repository somehow.

But I prefer to see if my second proposal is possible now as it's more interesting than go mod.

“More interesting” is a fine criterion for hobby projects, but it's not a good basis for inclusion in the Go toolchain: we try very hard to do make things “simple” rather than “interesting”.

A mechanical program to produce a go.mod file would add a huge amount of complexity — we (intentionally) do not execute any user-provided code during a go build.

Even environment substitution would add a lot of complexity in order to address a fairly niche use-case. Moreover, there already a lot of alternatives available, such as a local server with <meta> tags, a git post-checkout hook, or replace directives that point to symlinks that developers can customize.

I appreciate the thought you've put into this, but we won't be supporting these features in the go tool. (You are welcome to write whatever outside tooling you like to manage your own go.mod files, of course!)

@bcmills

• it requires to have real domain name and hosted meta tags plus potential bandwidth for transferring packages.

Last I checked a real domain name is $12/yr., and I'm not sure that you even need that: a reserved local IP address (even a loopback address to your local machine!) would suffice, as long as the <meta> tags point to someplace that you can resolve.

• not always do we need to configure a remote setting to import a package -- I think this is one of the reason why replace was implemented.

I'm not sure what you mean. If you're importing a package from the same repository, you can use relative replace directives within the module. If you're importing from a different repository, you need to be able to resolve that repository somehow.

For just building a project, do you think we (users) really need to buy a domain name and set a gopkg.in-like meta tags server to redirect import paths/versions. (If remember this right, a new version of meta tags requires the meta tags server to transfer the entire package/module via HTTPS.)

The old GOPATH approach was making go build dev so easy to setup. We just need to put any third party packages/module into GOPATH. It works well except that versioning/vendoring dependencies is difficult. And I don't think a meta-tags HTTP delegate server is making the
things easier.

It's nice to see the go.mod approach. It allows us to manage packages with the concept of module and avoid setting up the HTTP delegate server (meta-tags) to satisfy import. The meta-tags delegation is really a bad approach and you're suggesting me to use it.

But I prefer to see if my second proposal is possible now as it's more interesting than go mod.

“More interesting” is a fine criterion for hobby projects, but it's not a good basis for inclusion in the Go toolchain: we try very hard to do make things “simple” rather than “interesting”.

The second proposal is just another topic than theusing the directives which I'm very interesting to try. It's interesting because it's working very similar to the go test mechanism and it's possible to allow devs to achieve more than go.mod directives.

Regarding to the using directives proposal to go.mod, your only reason of rejection is that it's bringing more complexity to cmd/go. This is very interesting, I think I can understand if you just say "I'm not going to add it" instead of complexity. A little bit of improvement to go tool can benefit thousands and thousands of users. It's not an good excuse to reject because of it's too complex to implement. Lucky for us, Go is open source, so that we can hack it ourselves.

I proposed the using directives is just wishing that go tool could become better. Thanks for responding anyway, the communication helped me to learn and think more.

Thanks again.

For just building a project, do you think we (users) really need to buy a domain name and set a gopkg.in-like meta tags server to redirect import paths/versions.

For “just building a project”, I would expect that most users either have a domain name already or use a (free) code hosting service. (If #28835 is accepted, that would potentially enable some other workflows as well, but if you want to interact with a global namespace of modules, at some point you need a unique name within that namespace.)

It's not an good excuse to reject because of it's too complex to implement.

When I say it “would add a huge amount of complexity”, I mean it would be complex to use, not (just) complex to implement. The more features we add, the more features people have to understand in order to interact with the projects that use them.

When I say it “would add a huge amount of complexity”, I mean it would be complex to use, not (just) complex to implement. The more features we add, the more features people have to understand in order to interact with the projects that use them.

I would be agree with you that the more we add, the more complexity (of using it) it might be.

Comparing to the using directives and the real custom domain coding hosting server plus meta-tags, I don't believe that using is more complex to use. But I would be agree that a domain name being used as a package/module namespace is a good idea. So example.io/foo looks better to me than example/foo. But if you forces users to setup a real example.io code server in all cases, I insist that it's not easier than the using directives for go.mod.

Regarding to using the go_tooling.go approach as a replacement for go.mod, I always believe that it would be much easier to use a Go tooling API than a bunch of go.mod directives. As we're basically all Go devs, we all knows Go. So I believe that go_tooling.go wouldn't be more complex than writing directives in go.mod.

But I do believe that go_tooling.go should be much more complex to implement than the using directives for go.mod.

If I'm going to implement a go/tooling API, I should make use of all the internal funcs already implemented for go.mod. My basic idea was to populate the very common cmd/go/internal features into go/tooling, and use special Go files like xxx_test.go, go_tooling.go or anything else to allow Go editors/developers to interact with go tool. And I do believe that designing go/tooling API well is not easy but worthy.

I like this second approach because it's all in Go, no need to have extra grammars like go.mod. It's interesting because it would allows us (users/developers) to have unlimited ultra interaction with go tool.

For a summary:

• using Go code to replace any meta-language (or domain language) like go.mod - Go itself as the meta language
• to have a unlimited possibilities to extend go tool (including fixing the issues that using directive was trying to for go.mod)

I think these two principles make me feel interesting on this issue. As it's following the initial saying of Go that Go is making programming interesting again (comparing to C/C++ or other languages).