Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: get / mod: document pruning repos for "module source tree too big" error #32297

Closed
paulquerna-okta opened this issue May 29, 2019 · 11 comments
Closed

cmd/go: get / mod: document pruning repos for "module source tree too big" error #32297

paulquerna-okta opened this issue May 29, 2019 · 11 comments
Labels
Documentation FrozenDueToAge modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone
Go1.14

Comments

@paulquerna-okta
Copy link

What version of Go are you using (go version)?

go version go1.12.2 darwin/amd64

What did you do?

Hello Gophers. We are attempting to migrate several large proprietary Go ecosystems to using Go Modules. Previously we have used glide and would like to be on track with where the community is going.

When attempting to migrate repo alpha that depends on beta, we received the module source tree too big error. We have already migrated beta to Go modules in a development branch, which is the sha reference in the go get bellow.

This error also prevents commands like go mod vendor from working, and is seemingly hidden from showing up as an error there - it treats the module source tree too big error to imply the module import path is invalid, so some commands "kinda" work, but then we get later failures because the files weren't actually placed in vendor/.

But, you can see the direct error when attempting a go get of the repo:

$ go get github.com/okta/beta@157700ca4b7c2b57e46cde2a531ed3889833349a
go: finding github.com/okta/beta 157700ca4b7c2b57e46cde2a531ed3889833349a
go: downloading github.com/okta/beta v1.39.2
package github.com/okta/beta: unknown import path "github.com/okta/beta": module source tree too big

$ cd beta
$ du -sh .
1020M	.
$ du -sh .git
368M	.git

What did you expect to see?

go mod vendor and go get to work with large repos. Alternatively a flag or environment variable available to increase the limit would be viable for our environment.

What did you see instead?

The module source tree too big error.

Previous reports:
#25470
#29210

In the previous report, #29210 (comment) "Folks can file separate issues if they discover other repositories that don't fit." is the last comment, so here we are.
@agnivade
Copy link
Contributor

Can you provide more details as asked by @bcmills in #29210 ?

What are the file types that contribute to the repo size ? Can you provide a breakdown ? Is all of it coming entirely from .go source files which are required to build it ? If not, have you tried pruning them from being included in the module by placing another go.mod in the right directory ?

@agnivade agnivade added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 29, 2019
@paulquerna-okta
Copy link
Author

We were able to create go.mod files in a couple sub-directories to get us under the current limit. If the limit is going to stay, is there a place that can document this practice of splicing repos with creating a new go module under a directory? It seems to be the only solution being advised?

@agnivade agnivade removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jun 5, 2019
@agnivade
Copy link
Contributor

agnivade commented Jun 5, 2019

@bcmills @jayconrod @thepudds

@jayconrod
Copy link
Contributor

@paulquerna-okta Does the repo just contain source files and test data needed by go build and go test? Are there any large files that can be removed in tagged versions, as there were in #29210? So far, 500MB has been enough space for buildable code, but we'd like to understand if that's appropriate.

The purpose of the limit is to avoid making users download large artifacts (zip files) with files that aren't actually needed. Files that aren't part of the build (large resources, vendor directories, etc) may be removed in tagged versions to get below this limit. These files can usually still be on development branches though.

@jayconrod jayconrod added modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 5, 2019
@paulquerna-okta
Copy link
Author

We cannot just remove these files from "tagged" versions in git -- we are tagging a whole software project which combines multiple programming languages. You are right as far as the Go compiler / modules are concerned, there are many files that are not relevant. It's pretty common pattern to have a .ignore file, like .gitignore or .dockerignore when you want an ecosystem of tools to ignore things. Has an idea like that been explored for creating the Go Modules zip files?

This is really an intersection of Go Modules and two mono-repositories. Mono-repos like this are less common in open source, so I'm less sure this is an issue about users downloading large files. We're on the path to breaking up the mono-repos, but kinda ironically that path was to first get to Go Modules tooling to make it easier.

@bcmills
Copy link
Contributor

bcmills commented Jun 5, 2019

I agree that removing the files from tagged versions is not really a viable option.

We've been using go.mod files as the equivalent of .ignore, and at this point that would be very difficult to change: any change in the extracted contents of a module also changes its hash, and we currently have no way to vary that based on the version of the go tool (see #30369).

Right now the supported mechanism is to add go.mod files at the root of any directories that should be pruned out of the Go module. (It should be fine if they're empty, as long as they exist.)

@agnivade
Copy link
Contributor

agnivade commented Jun 5, 2019

@bcmills - Does it make sense to note this trick in the wiki ? Seems useful. Or is it already documented somewhere ?

@bcmills
Copy link
Contributor

bcmills commented Jun 5, 2019

@thepudds might know. I don't see it today.

@bcmills bcmills added Documentation NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 5, 2019
@bcmills bcmills added this to the Go1.14 milestone Jun 5, 2019
@agnivade agnivade changed the title cmd/go: get / mod: "module source tree too big" cmd/go: get / mod: document pruning repos for "module source tree too big" error Jun 15, 2019
@agnivade agnivade removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 15, 2019
@katiehockman
Copy link
Contributor

@bcmills @thepudds any updates/thoughts?

@katiehockman katiehockman added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 17, 2019
@thepudds
Copy link
Contributor

@katiehockman I added a short FAQ on this:

https://github.com/golang/go/wiki/Modules#can-an-additional-gomod-exclude-unnecessary-content-do-modules-have-the-equivalent-of-a-gitignore-file

@agnivade
Copy link
Contributor

Thanks @thepudds

@golang golang locked and limited conversation to collaborators Jun 17, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
Documentation FrozenDueToAge modules NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
7 participants
@jayconrod @agnivade @bcmills @gopherbot @thepudds @katiehockman @paulquerna-okta