Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: support local experiments with interdependent modules; then retire GOPATH #44347

Closed
ohir opened this issue Feb 17, 2021 · 19 comments

Comments

@ohir
Copy link

ohir commented Feb 17, 2021

Go "tinker mode" proposal.

Last updated: 2021/04/11

Rationale:

@cosban wrote: I need to be able to, without being forced to commit code that is not fully vetted, build and test our modifications. (@cosban)


@rsc wrote: GOPATH is holding back the ecosystem and the toolchain. It's time to retire it. [...] we're happy to listen. But GOPATH needs to go. (@rsc)


@bcmills wrote: Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules (@bcmills)

The main workaround at the moment is to add replace directives among the modules to be edited, but maintaining those directives is tedious and error-prone. @rogpeppe's gohack tool automates away some of the tedium, but doesn't seem to remove the risk of accidentally checking in a go.mod with what were intended to be local, temporary replacements.


Related: #26640, #37755, #26377, #25053.

If an environment variable named GOTINKER is defined and set to an absolute path to the existing location on the local filesystem, and the import path of an include can be found under $GOTINKER/ directory, then build commands treat the $GOTINKER/src/import/path as a final authoritative source of the import; foregoing both vendor/ and any go.mod directive perpeting to this import/path.

Ie. build commands like 'go build' and 'go test' will compile modules present in the $GOTINKER directory instead of accessing the network, local module cache, or vendor directory.

Under tinker mode GOBIN, GOCACHE, GOMODCACHE, and GOENV are bound to locations relative to the GOTINKER: $GOTINKER/bin, $GOTINKER/cache, $GOTINKER/pkg/mod, and $GOTINKER/goenv - respectively.

$GOTINKER tree should be populated by the user. For yet some time to come the last version of Go to support GOPATH can be used to ease this task, ie. GOROOT=/where/go1.16 GO111MODULE=off GOPATH="$GOTINKER" go1.16 get

GOTINKER path last element may start with an underscore character so experiments can be kept inside any project tree.

Security considerations

Both object code and the executable built under the tinker mode should not accidentally leak to the production environment. Ie. while objects are built and cached under $GOTINKER, then built executable MUST be amended (by the compiler) to refuse to run in an environment where GOTINKER is not set, or it is set but does not match the last element of the GOTINKER path that was compiled in.

Ie. "tinkered with" executable preserves the last part of the GOTINKER path then matches it to the last part of GOTINKER string where it runs. If these do not match, exectutable exits immediately with "Experimental but GOTINKER is not set or did not match" error message.


How GOTINKER workflow is different to GOPATH''s one?

GOPATH is all-or-nothing regarding versions. Ie. under GOPATH tools operate on code as-is. If we are about debugging or changing interdependent code, it is up to us to do proper checkouts of everything our — possibly big — app or service uses.

In proposed GOTINKER mode tools operate on modules as usual, so anything else but code we pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for us, nothing will drift apart or leak accidentally.


edits:

  • 2021/02/17 added "match compiled in GOTINKER with run one" in Security
  • 2021/02/18 be explicit about possibility to keep GOTINKER tree inside a module tree, eg. in a _tinker/ subdirectory.
  • 2021/04/11 compare GOPATH and GOTINKER workflows.
@gopherbot gopherbot added this to the Proposal milestone Feb 17, 2021
@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Feb 17, 2021
@ianlancetaylor
Copy link
Contributor

CC @bcmills @jayconrod

@jayconrod
Copy link
Contributor

cc @matloob @stamblerre @ianthehat

@complyue
Copy link

complyue commented Apr 9, 2021

I like this proposal more than other alternatives to the problem, hope it get more attention.

I used to have GOPATH=/globally-cached/go-deps:/personal-tinkering/go-devs:/team-tinkering/go-pkgs, works perfectly well with organization-wide shared filesystems.

And I hesitate to pickup further development of my rusted Go projects at this time, just because go.mod is the right way to go, but it still lacks sufficient support rival to my historical workflows with GOPATH like that.

@Merovius
Copy link
Contributor

Merovius commented Apr 9, 2021

I don't understand the significant difference between this proposal and just continuing to use GOPATH (and, I guess, giving it priority over modules). That is, GOTINKER seems to be almost identical in its interpretation to GOPATH. The most significant deviation seems to be the "refuse to run if GOTINKER is not set" part, correct? I don't see the benefit of that - to me, if the binary is able to run with some environment variable is set, even if it doesn't look at that, this seems to serve as conclusive proof that setting this environment variable doesn't provide any benefit.

@ohir
Copy link
Author

ohir commented Apr 9, 2021

That is, GOTINKER seems to be almost identical in its interpretation to GOPATH.

Yes it is. Yes — almost: the main and important difference is that it operates in module "units".

There are modules under the GOTINKER/src, not packages.
I can do cd‍ _tinker/src/import/path, clone/pull the source, mod-tidy it, then make changes at will. Repeat for other dependent modules pulled to _tinker/. When bug is fixed i can tag and push all the fixes to the modules repo simultanously.

Same for bootstraping and early experimenting with code structure.

When I am done with experiments, simple rm -rf _tinker && unset GOTINKER cleans up my workspace.

The most significant deviation seems to be the "refuse to run if GOTINKER is not set" part, correct?

The most significant deviation is described above. "Production fuse" is the second, along with builddirs moved inside GOTINKER. Both prevent experimental code leaks to the CI/CD pipelines. It shouldn't happen but it happens — and when it does, it hurts a lot.

if the binary is able to run with some environment variable is set, even if it doesn't look at that

It looks at, and if it sees wrong environment it gently refuses to do any work except saying why it won't run "here".

setting this environment variable doesn't provide any benefit.

To the developer, probably not much - she already has it set. But deployment people will be less nervous when asked to fire up a field test.

and just continuing to use GOPATH

GOPATH destiny, according to previous discussions, is to retire. GOTINKER needs the least code I could think of to get GOPATH benefits back. It boils down to:

  • check GOTINKER first; if set do:
  • in-memory: set GOENV (BIN CACHE)
  • scan the tree under GOTINKER
  • for every import/path found there and seen in go.mod too, recursive:
    • in-memory replace go.mod sourced entry relating to import/path with
    require import/path v0.0.0-unpublished
    replace import/path v0.0.0-unpublished => {{tinkerpath}}/src/import/path

...(skip code in vendor)...

@Merovius
Copy link
Contributor

Merovius commented Apr 10, 2021

There are modules under the GOTINKER/src, not packages.
I can do cd‍ _tinker/src/import/path, clone/pull the source, mod-tidy it, then make changes at will. Repeat for other dependent modules pulled to _tinker/. When bug is fixed i can tag and push all the fixes to the modules repo simultanously.

Apparently I'm still not getting it. This doesn't sound very different to me. For example, I currently have GOPATH=$HOME and under ~/src/github.com/Merovius/nbd I have a checkout of the module module github.com/Merovius/nbd - i.e. the module is in the import path, relative to GOPATH. If I set GO111MODULE=off, I can cd ~/src/github.com/Merovius/nbd, work in GOPATH mode (and thus use the rest of my ~/src, where the rest of the modules I work with is also checked out under the correcti mport paths), commit and push. All of that seems exactly the layout of what you describe when you say GO111MODULE=off go get … can be used for the setup of GOTINKER.

So, I really don't see the difference, TBQH. To me, that seems to be exactly the setup and workflow you are describing.

GOPATH destiny, according to previous discussions, is to retire.

Yes. That's why I'm confused. To me, what you are describing is almost exactly the (for now) still existing GOPATH based workflow - except for a renamed environment variable. If we think we need that workflow, it seems more straight-forward not to retire GOPATH . And if we do want to retire GOPATH, surely we wouldn't just re-introduce it under a different name.

@ohir
Copy link
Author

ohir commented Apr 10, 2021

If I set GO111MODULE=off, I can [...]

Minus that with 1.17 the GO111MODULE will always be on.

what you are describing is almost exactly the (for now) still existing GOPATH based workflow

It is supossed to be almost identical workflow - except for all tools being now in modules mode, so I can work only with, say, three repositories out of thirty or eighty. Modules that I do not plan to tinker with will come from cache (copied or hardlinked to the $GOTINKER/mod/cache). Also all GOPATH related code, esp. VCS one, may retire.

GOTINKER code will need do a little bit of magic to internal representation of go.mods read — then anything else works in module terms as if you'd edited-in all those require/replace directives into proper places by hand.

If we think we need that workflow

Yes "we" think we need. See Rationale at top.

@Merovius
Copy link
Contributor

Merovius commented Apr 10, 2021

If we think we need that workflow

Yes "we" think we need. See Rationale at top.

FTR, by "we" I mean "the Go project", not a specific subset of people. It certainly includes you and me and many others, many of whom likely agree with you and many of whom don't. It's possible for "the Go project" to come to a conclusion, even though some or many of its members disagree with it. The proposal process is how these conclusions are reached and are thus what decides what the "we" I was referring to "thinks".

So, let me rephrase my questions. If the outcome of this discussion is, that the workflow presented here is important enough to implement and given its extreme similarity (by design) to the existing GOPATH based workflow, wouldn't it be preferable to simply not retire GOPATH? And if, on the other hand, there are good, convincing reasons to retire GOPATH, why would those reasons not apply to this proposal? Surely, whether the environment variable is spelled GOPATH or GOTINKER does not affect those reasons?

Alternatively, this proposal is not as similar to GOPATH as it seems to me, which means I misunderstood something. That would certainly make these questions obsolete and provide justification for this proposal. Which is why I tried to understand these differences. So, is it actually the case that the workflow you describe is currently (we are not talking about go 1.17+) captured by setting GO111MODULE=off? If not, what are the differences? Your last answer "GOTINKER operates on modules, not packages" is confusing to me, as GOPATH seemingly contains exactly the same directory layout, containing exactly the same repositories with exactly the same files. But maybe I misunderstood something?

@complyue
Copy link

@Merovius As I understand it, one very important difference is that while GOPATH is an exclusive choice against go.mod, GOTINKER can work together with go.mod.

I for myself especially want the benefits of GOPROXY with go.mod on (to workaround GFW as one reason), I can't get those by choosing GOPATH, while GOTINKER is hopeful to provide equally ideal workflows like with GOPATH, and still have all modern & good things from go.mod.

@Merovius
Copy link
Contributor

Merovius commented Apr 10, 2021

@complyue

As I understand it, one very important difference is that while GOPATH is an exclusive choice against go.mod, GOTINKER can work together with go.mod.

What does that mean? Concretely? Like, presumably the intent is to actually use local modifications, ignoring versions specified in go.mod. Doesn't that just mean "ignore go.mod"? If there are stanzas of go.mod we don't want to ignore, couldn't we just teach the go tool to use them in "GOPATH-mode"?

FWIW, as far as I can tell so far, the only reason to call it GOTINKER is to avoid association with GOPATH. There is no (significant) technical difference, but rather "GOPATH is going away, but maybe, if we call it GOTINKER, it can stay". That's an uncharitable interpretation, though, which is why I'm trying to coax out the actual differences that are proposed here.

I for myself especially want the benefits of GOPROXY with go.mod on (to workaround GFW as one reason), I can't get those by choosing GOPATH

I don't understand why this would require a new environment variable. To me, this seems to simply mean downloading the zip and unpacking it in the right directory (same as gohack already does). The go tool, with GO111MODULE=off" does not care whether the code in $GOPATHis from a git repository, or a zip-file - AFAIK it even completely ignores the VCS aftergo get`.

@ohir
Copy link
Author

ohir commented Apr 10, 2021

FTR, by "we" I mean "the Go project", not a specific subset of people.

I meant specific subset, hence quotation marks.

So, let me rephrase my questions. If [...] the workflow presented here is important enough to implement [...] given its extreme similarity to the existing GOPATH based workflow, wouldn't it be preferable to simply not retire GOPATH?

Probably keeping GOPATH mode would be a least friction solution to the interdependent edits problem in the short term. But, after a period of being in denial, I now second Rob Pike's "GOPATH must go" statement — because the technical debt from keeping GOPATH workflow intact will IMO hurt the ecosystem in the long run. Modules with their clearly, automatically accounted and double-checked versioning mechanics are superior to GOPATH for any software that evolves fast, for long, or both.

That said, modules workflow has a blindspot where interdependent edits of two or more modules come with a huge footgun loaded: you need to be very careful editing your replace directives by hand, in many places, then you need to be very careful again when you need to clean your laboratory table after. Citing @bcmills again:

Many module issues and questions seem to center on editing, testing, and deploying multiple (possibly mutually-interdependent, possibly cyclic) modules.

(Just now /04.2021/ I see three discussions about "how to do it with modules" active on the golang-nuts list. Such questions emerge there almost every month since go1.11)

And if, on the other hand, there are good, convincing reasons to retire GOPATH.

GOPATH is all-or-nothing. When it is on, all used modules repos must go there, and proper (re being investigated code) versions must be checked out by hand. Then vendoring status of each must be consulted too. GOPATH mode is currently, ie. in modules filled world, good only for bootstraping new code, it is unusable for the bughunt purposes.

why would those reasons not apply to this proposal? [...]
Alternatively, this proposal is not as similar to GOPATH as it seems to me

This proposal keeps only good things from GOPATH mode — all sandboxed and all implemented by already existing mechanics of the modules.

as GOPATH seemingly contains exactly the same directory layout, containing exactly the same repositories with exactly the same files.

No, not the same files. And almost certainly not the same as being investigated production code uses.

But maybe I misunderstood something?
[...] what are the differences?

In GOPATH mode tools operate on repo's HEAD. It is up to you to do proper checkouts of everything your, possibly big, app or service uses.

In proposed GOTINKER mode tools operate on modules as usual, so anything else but repos you pulled under GOTINKER path is kept at version specified in respective go.mod. All bookkeeping is done for you, nothing will drift apart or leak accidentally.

@complyue
Copy link

@Merovius

Doesn't that just mean "ignore go.mod"?

Not entirely, I mean, I only want a few modules I put in local filesystem, to override what go tools would find according to go.mod, for vast of the rest dependencies, I would like them managed by the modern go.mod way.

If there are stanzas of go.mod we don't want to ignore, couldn't we just teach the go tool to use them in "GOPATH-mode"?

I'm not sure I understand you correctly, but the whole idea is to avoid things local-tinkering-only to go into go.mod, ideally it should always be what the general public is expected to see it.

this seems to simply mean downloading the zip and unpacking it in the right directory

I would like all the way go.mod is meant to work for hundreds of dependencies I base my projects on, my local WIP modules are relative very small in number. And GOPATH's way of go get is superseded by go.mod, isn't it?

@complyue
Copy link

@ohir My 1 cent for yet another proposal as I said in #44649 , I suggest a go.project or go.home file to manifest a local directory as farm of wip modules atop how go.mod works as is. I'm not sure you like the idea, and I don't have budget to propose it up, shame on me.

@ohir
Copy link
Author

ohir commented Apr 10, 2021

@complyue

I suggest a go.project or go.home file to manifest a local directory as farm of wip modules atop how go.mod works

"Project specific settings" already are addressed in the proposal by the means of local GOENV file inside the GOTINKER path. There is no need to add an another "special" file as a knob. Such file (like the go.mod changes) would be another box to check on the setup/cleanup lists.

You can use direnv to set GOTINKER for you. (Even now you can use direnv to set GOPRIVATE/GOPROXY in the WiP tree. Works well if you do not use much external modules).


Intended bootstrap workflow using GOTINKER is just to set it to the "project" dir. You then work inside src/import/paths below with all the bells and whistles of module mode. Ie. all external dependencies are resolved as usual, just your code import paths are replaced relative to each other internally by the build tools.

@complyue
Copy link

complyue commented Apr 10, 2021

@ohir

using GOTINKER is just to set it to the "project" dir.

To make myself clearer, I consider it would be another proposal without the GOTINKER env var, there go.project or go.home or go.farm serves as the alternative marker of the "project" dir, instead of an env var.

Plain filesystem structure is simpler than that plus env vars, and will be much easier to up-scale without env vars involved. Somewhat like NodeJS uses node_modules to mark localized dependencies, if go.farm file is used to mark a module farm, then we'll be able to support nested farms to reuse overriding modules in parent farm(s), while to place more specific overriding modules in the nested farm with higher precedence.

I mean I think that might be even better than GOTINKER proposed here, but I don't have the necessary funding (of time & energy) to pursuit that idea. So I support your proposal here as far as the situation goes at time of speaking.

@complyue
Copy link

I suggest GOTINKER may add support for colon (or semicolon) separated multiple directories as the good old GOPATH does.

Besides GOPATH, similar things have worked well with PYTHONPATH, JAVA_PATH etc., and the very original PATH for all these years, so why not with GOTINKER.

@ohir
Copy link
Author

ohir commented Apr 15, 2021

I suggest GOTINKER may add support for colon (or semicolon) separated multiple directories as the good old GOPATH does.

It was considered, then refuted. Rationale: GOTINKER is not meant to keep yet another way to build final version of a Go software as explicit tag-based versioning supported by the modules ecosystem is clearly superior to the multi-tree one.

(With GOPATH, the only way to keep semi-stable versioning of being used code was to keep multiple code trees — and staging versions there or keeping tinkering under vendor/ of a stub package [since v1.6]. This was as much error-prone as using go.mod replacing directives is now).

@rsc
Copy link
Contributor

rsc commented Jul 20, 2021

I believe that the overall use cases being addressed in this proposal are also handled by multi-module workspaces, #45713.

I suggest we close this as a duplicate of #45713 and work on making that proposal handle any use cases it is missing.

@ohir
Copy link
Author

ohir commented Jul 22, 2021

While I think that #45713 is much overengineered I accept that the work on its implementation already begun, so this much lighter proposal can't stand. Closing.

@ohir ohir closed this as completed Jul 22, 2021
@rsc rsc moved this from Incoming to Declined in Proposals (old) Dec 1, 2021
@golang golang locked and limited conversation to collaborators Jul 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

7 participants