Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

proposal: x/mod: new repo for module mechanics #31761

Closed
9 of 13 tasks
rsc opened this issue Apr 30, 2019 · 31 comments
Closed
9 of 13 tasks

proposal: x/mod: new repo for module mechanics #31761

rsc opened this issue Apr 30, 2019 · 31 comments
Labels
modules Proposal Proposal-Accepted
Milestone
Unreleased

Comments

@rsc
Copy link
Contributor

rsc commented Apr 30, 2019 •
edited by jayconrod

For the initial development of module support, we've kept everything in internal directories to make it easier to make changes as our understanding of what the pieces should look like becomes clearer. But now I think the basic pieces have become pretty clear, and we should think about exporting some packages to help other people who want to write tools working directly with module mechanics.

Note that I said "module mechanics". This is about supporting direct manipulation of modules themselves. It is not about supporting general development tools that need to do things like load packages in module mode. That use case, where modules are incidental rather than the focus, should remain in x/tools, specifically x/tools/go/packages. And the specific case of loading packages should still be done by invoking the go command, which will remain the single point of truth for those algorithms.

I propose to create a new repo golang.org/x/mod that will initially contain:

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

/cc @bcmills @jayconrod @ianthehat @myitcv @rogpeppe
@rsc rsc added this to the Proposal milestone Apr 30, 2019
@rsc rsc added the Proposal label Apr 30, 2019
@bcmills
Copy link
Contributor

bcmills commented Apr 30, 2019

There's an open issue that this proposal addresses (about exposing some of this stuff) but once again I cannot find it.

That's #28101.

@rsc rsc mentioned this issue Apr 30, 2019
Closed
@dmitshur
Copy link
Contributor

I propose to create a new repo golang.org/x/mod

Does this proposal include a plan for how the new module, modfile, semver, etc., packages will be kept in sync with the ones in cmd/go/internal/...?

@bcmills
Copy link
Contributor

bcmills commented Apr 30, 2019

I would assume that they'll be vendored in using go mod vendor.

That does imply that we'll have to have a clean (and perhaps smaller) API boundary between golang.org/x/mod and cmd/go/internal, but that's arguably a good thing anyway.

@rsc
Copy link
Contributor Author

rsc commented Apr 30, 2019

@dmitshur, yes they would be vendored into cmd/vendor using "go mod vendor".
All the ones I've chosen should already have clean API boundaries, and if not, we'd work on that.

@ianthehat
Copy link

I need quite a few of these in order to support mod files in editors. I have an experiment I have been working on that has them copied to an internal directory of x/tools, but it would be much nicer if I could go straight to using this repository instead.

@hyangah
Copy link
Contributor

hyangah commented May 3, 2019

Consider pseudo-version related logic as well
e.g. https://go.googlesource.com/go/+/refs/heads/master/src/cmd/go/internal/modfetch/pseudo.go

@jayconrod
Copy link
Contributor

The list of packages above looks good. I expect this will be useful for developing gorelease and a lot of other tools.

I notice that none of internal/modload, internal/modfetch, internal/modfetch/coderepo, internal/mvs are in the list. I think that's the right decision for now -- these are all pretty tightly coupled with cmd/go. It might be worth moving these to x/mod after some refactoring though. I think the proxy and discovery projects have run into use cases where query and download functionality would be useful, and the Go command doesn't do quite what they need.

@rsc
Copy link
Contributor Author

rsc commented May 7, 2019

Sentiment here seems overwhelmingly in favor. Accepting proposal.

@rsc rsc modified the milestones: Proposal, Go1.13 May 7, 2019
@rsc
Copy link
Contributor Author

rsc commented May 7, 2019

Obviously x/mod itself is "unreleased" but it will have effects in the main repo, so milestoned to Go 1.13.

@rsc
Copy link
Contributor Author

rsc commented May 10, 2019

@andybons, I created this repo. Can you take care of adding it to gopherbot, builders, etc.? Thanks.

@gopherbot
Copy link

Change https://golang.org/cl/176463 mentions this issue: sumdb/note: signed note formats for checksum database

@gopherbot
Copy link

Change https://golang.org/cl/176464 mentions this issue: sumdb/dirhash: directory tree hash algorithm

@gopherbot
Copy link

Change https://golang.org/cl/176465 mentions this issue: sumdb/storage: basic storage interface and tests

@gopherbot
Copy link

Change https://golang.org/cl/176466 mentions this issue: sumdb: client and server protocol for checksum database

@gopherbot
Copy link

Change https://golang.org/cl/176460 mentions this issue: semver: basic semver parsing package

@gopherbot
Copy link

Change https://golang.org/cl/176461 mentions this issue: module: basic module.Version type and support

@gopherbot
Copy link

Change https://golang.org/cl/176462 mentions this issue: sumdb/tlog: transparent log data structures and algorithms

@andybons
Copy link
Member

Everything is set up for x/mod (mirroring, gerritbot, golang.org/x/mod, etc.)

@gopherbot
Copy link

Change https://golang.org/cl/176639 mentions this issue: sumdb/gosumcheck: basic go.sum checker

@gopherbot
Copy link

Change https://golang.org/cl/176540 mentions this issue: proxy: basic proxy server protocol

@gopherbot
Copy link

Change https://golang.org/cl/176541 mentions this issue: gomodproxy: simple Go module proxy implementation

gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
@rsc
semver: basic semver parsing package
78b9ff6 
Copied from cmd/go/internal/semver.

For golang/go#31761.

Change-Id: I59e8a264f49a5d20ba1caa85b59c3177209a6ff5
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176460
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 13, 2019
@rsc
module: basic module.Version type and support
5d307ac 
Copied from cmd/go/internal/module and added Version.String method.

For golang/go#31761.

Change-Id: I2fbe224843378e68f0e5890dbc36766936a7e3ed
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176461
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@thepudds thepudds mentioned this issue May 13, 2019
Open
@dmitshur dmitshur mentioned this issue May 14, 2019
Closed
@dmitris
Copy link
Contributor

dmitris commented May 28, 2019

I'm looking for a published library with the types / structures listed in the output of go mod help edit - namely Module, GoMod, Require, and Replace, ideally with the String() methods attached? Would they be in the new library (x/mod)?

@dmitshur
Copy link
Contributor

@dmitris Yes, that will be the golang.org/x/mod/modfile package. It hasn’t been added to x/mod just yet.

gopherbot pushed a commit to golang/mod that referenced this issue May 31, 2019
@rsc
sumdb: client and server protocol for checksum database
a1091b6 
Copied from golang.org/x/exp/sumdb/internal/sumweb
with types cleaned up:

	Conn -> Client
	Client -> ClientOps
	Handler -> Server
	Server -> ServerOps

For golang/go#31761.

Change-Id: If0e004e6c9cab69c82de428810d67aba074aa843
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176466
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
gopherbot pushed a commit to golang/mod that referenced this issue May 31, 2019
@rsc
gosumcheck: basic go.sum checker
d829663 
Copied from golang.org/x/exp/sumdb/gosumcheck.

For golang/go#31761.

Change-Id: Ia2cf81372b8e2da9b879f3d737de014eb30ea34b
Reviewed-on: https://go-review.googlesource.com/c/mod/+/176639
Run-TryBot: Russ Cox <rsc@golang.org>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Hyang-Ah Hana Kim <hyangah@gmail.com>
@josharian josharian mentioned this issue Jun 21, 2019
Closed
@tsal
Copy link

tsal commented Jun 22, 2019

Do we know when that might happen? This would be very useful for some CI tooling.

@tsal
Copy link

tsal commented Jun 28, 2019

Oops! I missed that this was added to the Go1.13 milestone back in May!

@dmitshur dmitshur mentioned this issue Jun 29, 2019
Closed
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@radeksimko radeksimko mentioned this issue Aug 7, 2019
Merged
@ianthehat ianthehat mentioned this issue Sep 9, 2019
Closed
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@gopherbot
Copy link

Change https://golang.org/cl/202543 mentions this issue: modfile: copy from cmd/go/internal/modfile

gopherbot pushed a commit to golang/mod that referenced this issue Oct 22, 2019
modfile: copy from cmd/go/internal/modfile
8ecbfe1 
Copied from 4be6b4a73d (CL 202565). No changes other than import paths.

Updates golang/go#34924
Updates golang/go#31761

Change-Id: Ic25cb983f6641045fc24edf76953b06d4aa5cd43
Reviewed-on: https://go-review.googlesource.com/c/mod/+/202543
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/202698 mentions this issue: cmd/go: delete internal packages moved to x/mod

@gopherbot
Copy link

Change https://golang.org/cl/203537 mentions this issue: cmd/dist: support GOROOT vendoring

gopherbot pushed a commit that referenced this issue Oct 28, 2019
cmd/dist: support GOROOT vendoring
60f34f7 
In the second step of make.bash, cmd/dist builds cmd/go by invoking
the compiler, linker, and other tools directly on transitive
dependencies of cmd/go. Essentially, cmd/dist acts as a minimal
version of 'go install' when building go_toolchain.

Until now, cmd/go has had no transitive dependencies in vendor
directories. This changes in CL 202698, where several packages are
deleted and equivalent versions in golang.org/x/mod are used
instead. So this CL adds support to cmd/dist for vendor directories.

Updates #31761

Change-Id: Iab4cdc7e505069a8df296287d16fbaa871944955
Reviewed-on: https://go-review.googlesource.com/c/go/+/203537
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/204138 mentions this issue: sumdb: sync change from cmd/go internal package

@gopherbot
Copy link

Change https://golang.org/cl/204103 mentions this issue: cmd/go/internal/modfile: don't use cmd/internal/diff

gopherbot pushed a commit that referenced this issue Oct 29, 2019
cmd/go/internal/modfile: don't use cmd/internal/diff
9f93fd2 
This is a partial revert of CL 203218.

cmd/go/internal/modfile is about to be deleted and replaced with
golang.org/x/mod/modfile in CL 202698. cmd/internal/diff is not
visible from golang.org/x/mod/modfile, and it doesn't make sense to
extract it into a new package there.

Updates #31761

Change-Id: I3bbbc4cae81120020e1092c1138524729530b415
Reviewed-on: https://go-review.googlesource.com/c/go/+/204103
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
gopherbot pushed a commit to golang/mod that referenced this issue Oct 29, 2019
sumdb: sync change from cmd/go internal package
18c3998 
Copy change in CL 203607 from cmd/go/internal/sumdb to
golang.org/x/mod/sumdb. This just removed an unused parameter. These
packages should be identical (except for imports) before
cmd/go/internal/sumdb is deleted in CL 202698.

Updates golang/go#31761

Change-Id: I1ce96d66bc71ef5c9515b68811049c47dce97765
Reviewed-on: https://go-review.googlesource.com/c/mod/+/204138
Run-TryBot: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
gopherbot pushed a commit that referenced this issue Oct 29, 2019
cmd/go: delete internal packages moved to x/mod
1fb7d54 
This change deletes several internal packages, replaces imports to
them with the equivalent golang.org/x/mod packages, updates x/mod, and
re-runs 'go mod vendor'.

Packages are replaced as follows:

cmd/go/internal/modfile → golang.org/x/mod/modfile
cmd/go/internal/module → golang.org/x/mod/module
cmd/go/internal/semver → golang.org/x/mod/semver
cmd/go/internal/sumdb → golang.org/x/mod/sumdb
cmd/go/internal/dirhash → golang.org/x/mod/sumdb/dirhash
cmd/go/internal/note → golang.org/x/mod/sumdb/note
cmd/go/internal/tlog → golang.org/x/mod/sumdb/tlog

Updates #31761
Fixes #34924

Change-Id: Ie3bf677bb0be49af969f654a0214243a6547eb57
Reviewed-on: https://go-review.googlesource.com/c/go/+/202698
Run-TryBot: Jay Conrod <jayconrod@google.com>
TryBot-Result: Gobot Gobot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@rsc
Copy link
Contributor Author

rsc commented Aug 5, 2022

This is done.

@rsc rsc closed this as completed Aug 5, 2022
@dmitshur dmitshur modified the milestones: Backlog, Unreleased Jun 4, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
modules Proposal Proposal-Accepted
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests