Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: allow 'go mod tidy' to proceed despite errors #26603

Closed
bcmills opened this issue Jul 25, 2018 · 10 comments
Closed

cmd/go: allow 'go mod tidy' to proceed despite errors #26603

bcmills opened this issue Jul 25, 2018 · 10 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.16

Comments

@bcmills
Copy link
Contributor

bcmills commented Jul 25, 2018

When adding module definitions to existing repositories, we may (often?) find that one or more existing packages do not actually build, or that some import cannot be resolved due to a bug (e.g., #26602).

Failure to build is especially likely for files with rarely-used build constraints, but rarely-used constraints are exactly the space that go mod -sync scans.

If the user needs to intervene to fix something, it would be nice if they could generate the rest of the module definition before they start trying to troubleshoot one or two packages that still won't build. For example, here is the output of go mod -sync for golang.org/x/build: it's a huge dependency graph, and seems to be held back by one missing import.

~/src/golang.org/x/build$ go mod -sync
go: finding golang.org/x/oauth2 latest
go: finding golang.org/x/time/rate latest
go: finding google.golang.org/api/googleapi latest
go: finding golang.org/x/time latest
go: finding google.golang.org/api latest
go: finding golang.org/x/sync/errgroup latest
go: finding google.golang.org/api/oauth2/v2 latest
go: finding golang.org/x/sync latest
go: finding google.golang.org/api/oauth2 latest
go: finding github.com/golang/protobuf/ptypes/timestamp latest
go: finding github.com/golang/protobuf/ptypes latest
go: finding google.golang.org/api/option latest
go: finding google.golang.org/genproto/googleapis/api/label latest
go: finding google.golang.org/genproto/googleapis/api latest
go: finding google.golang.org/genproto/googleapis latest
go: finding google.golang.org/genproto latest
go: finding grpc.go4.org latest
go: finding google.golang.org/appengine/urlfetch latest
go: finding google.golang.org/appengine/datastore latest
go: finding go4.org/types latest
go: finding go4.org/strutil latest
go: finding google.golang.org/api/deploymentmanager/v2 latest
go: finding go4.org latest
go: finding google.golang.org/api/deploymentmanager latest
go: finding github.com/google/go-github/github latest
go: finding google.golang.org/appengine/memcache latest
go: finding golang.org/x/perf/storage latest
go: finding github.com/gregjones/httpcache latest
go: finding golang.org/x/perf latest
go: finding google.golang.org/genproto/googleapis/monitoring/v3 latest
go: finding google.golang.org/appengine/delay latest
go: finding google.golang.org/genproto/googleapis/monitoring latest
go: finding golang.org/x/crypto/ssh latest
go: finding golang.org/x/crypto latest
go: finding golang.org/x/crypto/acme/autocert latest
go: finding golang.org/x/net/http2 latest
go: finding golang.org/x/crypto/acme latest
go: finding golang.org/x/net latest
go: finding github.com/davecgh/go-spew/spew latest
go: finding github.com/tarm/serial latest
go: finding google.golang.org/appengine/log latest
go: finding github.com/shurcooL/httpgzip latest
go: finding golang.org/x/sys/unix latest
go: finding google.golang.org/api/iterator latest
go: finding golang.org/x/sys latest
go: finding github.com/golang/protobuf/proto latest
go: finding github.com/shurcooL/issues latest
go: finding github.com/shurcooL/issues/maintner latest
go: finding go4.org/syncutil latest
go: finding google.golang.org/api/compute latest
go: finding github.com/jellevandenhooff/dkim latest
go: finding golang.org/x/tools/go/vcs latest
go: finding github.com/google/go-cmp/cmp latest
go: finding golang.org/x/tools/go latest
go: finding github.com/bradfitz/go-smtpd/smtpd latest
go: finding golang.org/x/oauth2/google latest
go: finding github.com/bradfitz/go-smtpd latest
go: finding golang.org/x/tools latest
go: finding golang.org/x/net/context/ctxhttp latest
go: finding google.golang.org/api/container latest
go: finding golang.org/x/net/context latest
go: finding golang.org/x/net/nettest latest
go: finding google.golang.org/genproto/googleapis/api/metric latest
go: finding github.com/shurcooL/issuesapp latest
go: finding github.com/coreos/go-systemd/activation latest
go: finding github.com/coreos/go-systemd/daemon latest
go: finding github.com/coreos/go-systemd latest
go: finding github.com/shurcooL/gofontwoff latest
go: finding cloud.google.com/go/monitoring/apiv3 latest
go: finding cloud.google.com/go/storage latest
go: finding cloud.google.com/go/bigquery latest
go: finding cloud.google.com/go/errorreporting latest
go: finding cloud.google.com/go/datastore latest
go: finding cloud.google.com/go/monitoring latest
go: finding cloud.google.com/go/compute/metadata latest
go: finding cloud.google.com/go/compute latest
go: finding github.com/shurcooL/github_flavored_markdown latest
go: finding github.com/shurcooL/htmlg latest
go: finding github.com/shurcooL/users latest
go: finding github.com/shurcooL/octicon latest
go: finding github.com/shurcooL/reactions/component latest
go: finding github.com/shurcooL/notifications latest
go: finding github.com/dustin/go-humanize latest
go: finding github.com/shurcooL/reactions latest
go: finding google.golang.org/grpc/metadata latest
go: finding google.golang.org/grpc/codes latest
go: finding github.com/shurcooL/httpfs/html/vfstemplate latest
go: finding github.com/shurcooL/httpfs/html latest
go: finding github.com/google/go-querystring/query latest
go: finding github.com/shurcooL/httpfs latest
go: finding google.golang.org/grpc/status latest
go: finding github.com/shurcooL/httpfs/union latest
go: finding github.com/google/go-querystring latest
go: finding go.opencensus.io/plugin/ocgrpc latest
go: finding go.opencensus.io/stats/view latest
go: finding go.opencensus.io/trace latest
go: finding golang.org/x/text/unicode/norm latest
go: finding go.opencensus.io/plugin latest
go: finding go.opencensus.io/stats latest
go: finding github.com/shurcooL/github_flavored_markdown/gfmstyle latest
go: finding golang.org/x/text/unicode latest
go: finding google.golang.org/grpc/credentials latest
go: finding google.golang.org/grpc/credentials/oauth latest
go: finding google.golang.org/grpc/naming latest
go: finding github.com/shurcooL/events latest
go: finding github.com/shurcooL/events/event latest
go: finding golang.org/x/text/unicode/bidi latest
go: finding go.opencensus.io/plugin/ochttp latest
go: finding golang.org/x/text/secure/bidirule latest
go: finding github.com/anmitsu/go-shlex latest
go: finding github.com/shurcooL/webdavfs/vfsutil latest
go: finding go.opencensus.io/exporter/stackdriver/propagation latest
go: finding go.opencensus.io/exporter/stackdriver latest
go: finding github.com/shurcooL/webdavfs latest
go: finding go.opencensus.io/exporter latest
go: finding golang.org/x/text/secure latest
go: finding github.com/shurcooL/httperror latest
go: finding github.com/shurcooL/go/gopherjs_http latest
go: finding dmitri.shuralyov.com/state latest
go: finding github.com/shurcooL/go latest
go: finding github.com/google/martian/httpspec latest
go: finding dmitri.shuralyov.com/html/belt latest
go: import "golang.org/x/build/autocertcache" ->
	import "cloud.google.com/go/storage" ->
	test ->
	import "cloud.google.com/go/httpreplay" ->
	import "cloud.google.com/go/httpreplay/internal/proxy" ->
	import "github.com/google/martian/httpspec": cannot find module providing package github.com/google/martian/httpspec
@bcmills bcmills added this to the Go1.12 milestone Jul 25, 2018
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 25, 2018
@bcmills bcmills changed the title cmd/go: allow 'go mod -sync' to proceed despite errors cmd/go: allow 'go mod tidy' to proceed despite errors Oct 24, 2018
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Oct 24, 2018
@bcmills bcmills mentioned this issue Feb 27, 2019
Closed
@andybons andybons modified the milestones: Go1.13, Go1.14 Jul 8, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
@rogpeppe
Copy link
Contributor

There's a significant example of this that I encountered recently when using TinyGo. TinyGo has some additional packages in its standard library (for example, the "machine" package). Because those packages don't exist in the regular Go standard library, go mod tidy just gives up, which means that it's not possible to use it to tidy up, despite that other Go module-related commands can be used OK.

Here's a testscript example to demonstrate the issue:

go mod tidy

-- go.mod --
module example

go 1.15
-- main.go --
package main

import _ "machine"

func main() {
}

The go command fails with this error:

example imports
	machine: package machine is not in GOROOT (/home/rog/go/src/machine)

@bcmills
Copy link
Contributor Author

bcmills commented Jul 31, 2020

@rogpeppe, do the TinyGo source files have a particular build constraint? If so, the narrower fix described in #40067 (comment) might be more appropriate.

@rogpeppe
Copy link
Contributor

I showed a complete example above. No build constraints involved, I'm afraid.

@bcmills
Copy link
Contributor Author

bcmills commented Aug 1, 2020

Then I would argue the defect in that example is in TinyGo, not cmd/go. A program that expects a different version of the standard library should have a distinguished build tag (as we do for mainline Go releases) or a distinguished go version in the go.mod file.

If the program by all accounts indicates that it expects to be compiled in a standard go 1.15 environment, then the standard go command should diagnose issues that would prevent its dependencies from being loaded in a standard go 1.15 environment. (Build constraints exist for a reason.)

@rogpeppe
Copy link
Contributor

rogpeppe commented Aug 1, 2020

FWIW there's no problem with using the standard go command for this code (I can happily go get etc) but only go mod tidy as far as I can tell.

Since one of the aims of go mod tidy is to evaluate all Go code regardless of build tag, I don't really see why a build tag should affect things much (and it doesn't).

For my particular problem, it seems to me that go mod tidy could see the stdlib import (easy to tell because the first element of the path has no dots) and ignore the error because no stdlib import can ever imply an external dependency.

@bcmills
Copy link
Contributor Author

bcmills commented Aug 3, 2020

no stdlib import can ever imply an external dependency

Unfortunately, that does not hold in the presence of replace directives. (In particular, the missing import could be one that the user intended — but forgot — to shim in with a replace.)

@rogpeppe
Copy link
Contributor

rogpeppe commented Aug 3, 2020

Unfortunately, that does not hold in the presence of replace directives

You can use a replace directive to replace a stdlib package? Interesting. I had no idea.

@bcmills
Copy link
Contributor Author

bcmills commented Aug 3, 2020

You can't use it to replace a package that actually exists in the standard library, but you can use it to slot in a module whose path would normally be reserved for the standard library (just like you can do by manually poking directories into a GOPATH/src tree).

@gopherbot
Copy link

Change https://golang.org/cl/255960 mentions this issue: cmd/go: add a '-e' flag to 'mod tidy' and 'mod vendor'

@bcmills bcmills modified the milestones: Backlog, Go1.16 Sep 18, 2020
@bcmills bcmills self-assigned this Sep 18, 2020
@gopherbot
Copy link

Change https://golang.org/cl/287413 mentions this issue: content/static/doc: document -e flag for tidy and vendor

gopherbot pushed a commit to golang/website that referenced this issue Feb 16, 2021
_content/doc: document -e flag for tidy and vendor
b03180c 
For golang/go#26603

Change-Id: I4bb1138ebda5ded1be30ead19bd6b282eedc3265
Reviewed-on: https://go-review.googlesource.com/c/website/+/287413
Trust: Jay Conrod <jayconrod@google.com>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@golang golang locked and limited conversation to collaborators Jan 27, 2022
@rsc rsc unassigned bcmills Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.16
Development

No branches or pull requests
5 participants
@rogpeppe @rsc @andybons @bcmills @gopherbot