cmd/go: `go list -e -m -json all` to start displaying hashes of modules #52792

HakanSunay · 2022-05-09T15:58:24Z

I came to notice that the go tool does not list module hash sums when the list -e -m -json all command is invoked.

$ go mod init hash

$ go get github.com/vmware/govmomi@latest                                                                                                                         ─╯
go: downloading github.com/vmware/govmomi v0.28.0
go: added github.com/vmware/govmomi v0.28.0

$ go list -e -m -json all
{
	"Path": "hash",
	"Main": true,
	"Dir": "/Users/hhalil/test",
	"GoMod": "/Users/hhalil/test/go.mod",
	"GoVersion": "1.18"
}
{
	"Path": "github.com/a8m/tree",
	"Version": "v0.0.0-20210115125333-10a5fd5b637d",
	"Time": "2021-01-15T12:53:33Z",
	"Indirect": true
}
{
	"Path": "github.com/dougm/pretty",
	"Version": "v0.0.0-20171025230240-2ee9d7453c02",
	"Time": "2017-10-25T23:02:40Z",
	"Indirect": true
}
{
	"Path": "github.com/google/uuid",
	"Version": "v1.3.0",
	"Time": "2021-07-12T22:33:52Z",
	"Indirect": true
}
{
	"Path": "github.com/kr/pretty",
	"Version": "v0.3.0",
	"Time": "2020-11-24T22:22:38Z",
	"Indirect": true
}
{
	"Path": "github.com/kr/text",
	"Version": "v0.2.0",
	"Time": "2020-02-14T20:31:06Z",
	"Indirect": true
}
{
	"Path": "github.com/rasky/go-xdr",
	"Version": "v0.0.0-20170217172119-4930550ba2e2",
	"Time": "2017-02-17T17:21:19Z",
	"Indirect": true
}
{
	"Path": "github.com/vmware/govmomi",
	"Version": "v0.28.0",
	"Time": "2022-04-27T15:48:05Z",
	"Indirect": true,
	"Dir": "/Users/hhalil/go/pkg/mod/github.com/vmware/govmomi@v0.28.0",
	"GoMod": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.mod",
	"GoVersion": "1.17"
}
{
	"Path": "github.com/vmware/vmw-guestinfo",
	"Version": "v0.0.0-20170707015358-25eff159a728",
	"Time": "2017-07-07T01:53:58Z",
	"Indirect": true
}

That is not the case with go mod download -json

$ go mod download -json                                                                                                                                         ─╯
{
	"Path": "github.com/vmware/govmomi",
	"Version": "v0.28.0",
	"Info": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.info",
	"GoMod": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.mod",
	"Zip": "/Users/hhalil/go/pkg/mod/cache/download/github.com/vmware/govmomi/@v/v0.28.0.zip",
	"Dir": "/Users/hhalil/go/pkg/mod/github.com/vmware/govmomi@v0.28.0",
	"Sum": "h1:VgeQ/Rvz79U9G8QIKLdgpsN9AndHJL+5iMJLgYIrBGI=",
	"GoModSum": "h1:F7adsVewLNHsW/IIm7ziFURaXDaHEwcc+ym4r3INMdY="
}

The proposal here is to extend the underlying data structures of go list -m to present information such as Sum and GoModSum, just like go mod download -json. By definition, "The -m flag causes list to list modules instead of packages.", therefore I believe it makes sense for the module data to include hash information.

Also, it would be useful if go list -m can also function in workspace mode, meaning that it would present the hashes of the modules after MVS has been applied to all of them, but that is conditional logic on GOWORK.

The target struct to modify might be this:

go/src/cmd/go/internal/modinfo/info.go

Lines 12 to 27 in 53f1312

    
           type ModulePublic struct { 
        
           	Path       string        `json:",omitempty"` // module path 
        
           	Version    string        `json:",omitempty"` // module version 
        
           	Versions   []string      `json:",omitempty"` // available module versions 
        
           	Replace    *ModulePublic `json:",omitempty"` // replaced by this module 
        
           	Time       *time.Time    `json:",omitempty"` // time version was created 
        
           	Update     *ModulePublic `json:",omitempty"` // available update (with -u) 
        
           	Main       bool          `json:",omitempty"` // is this the main module? 
        
           	Indirect   bool          `json:",omitempty"` // module is only indirectly needed by main module 
        
           	Dir        string        `json:",omitempty"` // directory holding local copy of files, if any 
        
           	GoMod      string        `json:",omitempty"` // path to go.mod file describing module, if any 
        
           	GoVersion  string        `json:",omitempty"` // go version used in module 
        
           	Retracted  []string      `json:",omitempty"` // retraction information, if any (with -retracted or -u) 
        
           	Deprecated string        `json:",omitempty"` // deprecation message, if any (with -u) 
        
           	Error      *ModuleError  `json:",omitempty"` // error loading module 
        
           }

... and its implementation of the Stringer interface:

go/src/cmd/go/internal/modinfo/info.go

Lines 33 to 65 in 53f1312

    
           func (m *ModulePublic) String() string { 
        
           	s := m.Path 
        
           	versionString := func(mm *ModulePublic) string { 
        
           		v := mm.Version 
        
           		if len(mm.Retracted) == 0 { 
        
           			return v 
        
           		} 
        
           		return v + " (retracted)" 
        
           	} 
        
           	if m.Version != "" { 
        
           		s += " " + versionString(m) 
        
           		if m.Update != nil { 
        
           			s += " [" + versionString(m.Update) + "]" 
        
           		} 
        
           	} 
        
           	if m.Deprecated != "" { 
        
           		s += " (deprecated)" 
        
           	} 
        
           	if m.Replace != nil { 
        
           		s += " => " + m.Replace.Path 
        
           		if m.Replace.Version != "" { 
        
           			s += " " + versionString(m.Replace) 
        
           			if m.Replace.Update != nil { 
        
           				s += " [" + versionString(m.Replace.Update) + "]" 
        
           			} 
        
           		} 
        
           		if m.Replace.Deprecated != "" { 
        
           			s += " (deprecated)" 
        
           		} 
        
           	} 
        
           	return s 
        
           }

The text was updated successfully, but these errors were encountered:

HakanSunay · 2022-05-09T15:59:05Z

cc: @bcmills

With this change gazelle can invoke `update-repos` by processing `go.work` files. The basis of the change is a plain copy-paste of `update.go` into `work.go`. However, since `go list` does not show module sums, the whole process is delegated to `go mod download`. golang/go#52792 was created to track the above. The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary. A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked. How to invoke: $ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True

rsc · 2022-05-11T17:47:49Z

GoModSum would at least need to be a []string. When we do a transition to a new algorithm there will be multiple checksums.

rsc · 2022-05-11T18:02:51Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

rsc · 2022-05-18T17:19:14Z

/cc @bcmills @matloob

rsc · 2022-05-31T17:38:05Z

Looks like we dropped the ball on go mod download -json about string vs []string. Let's keep GoModSum a single string, and maybe in the future we'll have to add GoModSums listing all of them.

bcmills · 2022-05-31T17:38:48Z

SGTM. For reference. the go mod download -json struct definition is documented here:
https://pkg.go.dev/cmd/go#hdr-Download_modules_to_local_cache

bcmills · 2022-05-31T18:10:16Z

So this would be two new fields for the Module definition for go list, paralleling the Module definition for go mod download. The new definition for go list would then be:

type Module struct {
    Path      string       // module path
    Version   string       // module version
    Versions  []string     // available module versions (with -versions)
    Replace   *Module      // replaced by this module
    Time      *time.Time   // time version was created
    Update    *Module      // available update, if any (with -u)
    Main      bool         // is this the main module?
    Indirect  bool         // is this module only an indirect dependency of main module?
    Dir       string       // directory holding files for this module, if any
    GoMod     string       // path to go.mod file used when loading this module, if any
    GoVersion string       // go version used in module
    Retracted string       // retraction information, if any (with -retracted or -u)
    Error     *ModuleError // error loading module
    Sum       string       // checksum for path, version (as in go.sum)
    GoModSum  string       // checksum for go.mod (as in go.sum)
}

(Note that this would leave at least a couple of fields present in go mod download but absent from go list: namely, the Info and Zip fields.)

rsc · 2022-06-01T18:04:50Z

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

rsc · 2022-06-08T18:00:20Z

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

With this change gazelle can invoke `update-repos` by processing `go.work` files. The basis of the change is a plain copy-paste of `update.go` into `work.go`. However, since `go list` does not show module sums, the whole process is delegated to `go mod download`. golang/go#52792 was created to track the above. The standard go tooling in 1.18 can work with `go.work` files so not many changes were actually necessary. A smoke test is also added to `update_import_test.go` to verify that the latest version of govmomi and rest is always picked. How to invoke: $ gazelle update-repos -from_file=go.work -to_macro=deps.bzl%go_deps -prune=True

gopherbot · 2023-11-03T04:31:21Z

Change https://go.dev/cl/539575 mentions this issue: cmd/go/internal/modinfo: show Sum/GoModSum when listing modules

gopherbot · 2024-02-08T18:49:48Z

Change https://go.dev/cl/562775 mentions this issue: cmd/go: show Sum/GoModSum when listing modules

Fixes golang#52792 Tested: Ran go test cmd/go Change-Id: Ib7006256f4dca9e9fbfce266c00253c69595d6ab Cq-Include-Trybots: luci.golang.try:gotip-linux-amd64-longtest,gotip-windows-amd64-longtest Reviewed-on: https://go-review.googlesource.com/c/go/+/562775 Reviewed-by: Bryan Mills <bcmills@google.com> LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>

HakanSunay added the Proposal label May 9, 2022

gopherbot added this to the Proposal milestone May 9, 2022

bcmills added GoCommand cmd/go modules labels May 9, 2022

HakanSunay mentioned this issue May 9, 2022

Implement very minimalistic support for go workspaces bazelbuild/bazel-gazelle#1250

Merged

ianlancetaylor added this to Incoming in Proposals (old) May 9, 2022

rsc moved this from Incoming to Active in Proposals (old) May 11, 2022

rsc moved this from Active to Likely Accept in Proposals (old) Jun 1, 2022

rsc added the Proposal-FinalCommentPeriod label Jun 1, 2022

rsc moved this from Likely Accept to Accepted in Proposals (old) Jun 8, 2022

rsc changed the title ~~proposal: cmd/go: go list -e -m -json all to start displaying hashes of modules~~ cmd/go: go list -e -m -json all to start displaying hashes of modules Jun 8, 2022

rsc added the Proposal-Accepted label Jun 8, 2022

rsc modified the milestones: Proposal, Backlog Jun 8, 2022

rsc removed the Proposal-FinalCommentPeriod label Jul 26, 2023

bcmills added the help wanted label Sep 7, 2023

dmitshur modified the milestones: Backlog, Go1.22 Nov 17, 2023

gopherbot modified the milestones: Go1.22, Go1.23 Feb 6, 2024

bcmills mentioned this issue Feb 8, 2024

cmd/go: build with vendor depends on modcache for checksums #46400

Closed

dmitshur added the FixPending Issues that have a fix which has not yet been reviewed or submitted. label Feb 8, 2024

samthanawalla self-assigned this Feb 15, 2024

gopherbot closed this as completed in b2a169b Feb 16, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: `go list -e -m -json all` to start displaying hashes of modules #52792

cmd/go: `go list -e -m -json all` to start displaying hashes of modules #52792

HakanSunay commented May 9, 2022

HakanSunay commented May 9, 2022

rsc commented May 11, 2022

rsc commented May 11, 2022

rsc commented May 18, 2022

rsc commented May 31, 2022

bcmills commented May 31, 2022

bcmills commented May 31, 2022 •

edited

rsc commented Jun 1, 2022

rsc commented Jun 8, 2022

gopherbot commented Nov 3, 2023

gopherbot commented Feb 8, 2024

cmd/go: go list -e -m -json all to start displaying hashes of modules #52792

cmd/go: go list -e -m -json all to start displaying hashes of modules #52792

Comments

HakanSunay commented May 9, 2022

HakanSunay commented May 9, 2022

rsc commented May 11, 2022

rsc commented May 11, 2022

rsc commented May 18, 2022

rsc commented May 31, 2022

bcmills commented May 31, 2022

bcmills commented May 31, 2022 • edited

rsc commented Jun 1, 2022

rsc commented Jun 8, 2022

gopherbot commented Nov 3, 2023

gopherbot commented Feb 8, 2024

cmd/go: `go list -e -m -json all` to start displaying hashes of modules #52792

cmd/go: `go list -e -m -json all` to start displaying hashes of modules #52792

bcmills commented May 31, 2022 •

edited