Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: go get fails when fetching large repo #27646

Closed
ryandbump opened this issue Sep 12, 2018 · 6 comments
Closed

cmd/go: go get fails when fetching large repo #27646

ryandbump opened this issue Sep 12, 2018 · 6 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Go1.13

Comments

@ryandbump
Copy link

What version of Go are you using (go version)?

go version go1.11 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env 
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/ryanbump/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/ryanbump/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/Cellar/go/1.11/libexec"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.11/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/ryanbump/code/devolate/test-oversize/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/fn/lp9kz_ms6k353f9ml2j49rlw0000gn/T/go-build776699926=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

I created a repo github.com/ryandbump/test-oversize to recreate an issue I ran into when using go get. Thetest-oversize repo has 600M of junk txt files in it to increase it's size. There are also a few branches (junk-580M, junk-560M, etc.) that have different amounts of these junk text files to allow for testing with variable size repo.

In a module aware project run the following steps to reproduce.

  • run go get github.com/ryandbump/test-oversize
  • run go get for each of the different size branches.

What did you expect to see?

I expected to see the package downloaded and cached for use with my project.

What did you see instead?

An error was raised that implied there was a GOPATH issue despite using go get in a module aware context. The command would only succeed once the repo was below 500M in size.

$ go get github.com/ryandbump/test-oversize
go: finding github.com/ryandbump/test-oversize latest
go: downloading github.com/ryandbump/test-oversize v0.0.0-20180912214615-996567b1f368
go get: no install location for directory  outside GOPATH
	For more details see: 'go help gopath'
@agnivade
Copy link
Contributor

@bcmills @myitcv

@agnivade agnivade added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 13, 2018
@agnivade agnivade added this to the Go1.12 milestone Sep 13, 2018
@myitcv
Copy link
Member

myitcv commented Sep 13, 2018

@ryandbump please can you report the output with -x added?

@ryandbump
Copy link
Author 

$ go get -x github.com/ryandbump/test-oversize
# /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869 for git2 https://github.com/ryandbump/test-oversize
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git ls-remote -q https://github.com/ryandbump/test-oversize
0.296s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git ls-remote -q https://github.com/ryandbump/test-oversize
go: finding github.com/ryandbump/test-oversize latest
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368c1a266b531d0dbbdf78423702b06
0.027s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368c1a266b531d0dbbdf78423702b06
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
0.019s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git tag -l
0.016s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git tag -l
go: downloading github.com/ryandbump/test-oversize v0.0.0-20180912214615-996567b1f368
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368
0.011s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c log.showsignature=false log -n1 '--format=format:%H %ct %D' 996567b1f368
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
0.011s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git cat-file blob 996567b1f368c1a266b531d0dbbdf78423702b06:go.mod
cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 996567b1f368c1a266b531d0dbbdf78423702b06
3.225s # cd /Users/ryanbump/go/pkg/mod/cache/vcs/6d6322ff4148cd25804c23b7a0d9203f2ab1b2e7468d2f12c3b9b65900cf8869; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ 996567b1f368c1a266b531d0dbbdf78423702b06
go get: no install location for directory  outside GOPATH
	For more details see: 'go help gopath'

@hyangah
Copy link
Contributor

hyangah commented Oct 4, 2018

go/src/cmd/go/internal/modfetch/codehost/codehost.go

Line 30 in 8081862

MaxZipFile = 500 << 20 // maximum size of downloaded zip file

The error message is not very helpful. It seems to be reached the internal limit.

@bcmills
Copy link
Contributor

bcmills commented Oct 23, 2018

I created a repo github.com/ryandbump/test-oversize to recreate an issue I ran into when using go get.

What is the real issue you ran into? (Did the Go source files and testdata for the module overrun the 500MiB limit, or was our git command producing a zip archive containing irrelevant data?)

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 23, 2018
@bcmills bcmills modified the milestones: Go1.12, Go1.13 Nov 13, 2018
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@golang golang locked and limited conversation to collaborators Nov 23, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Go1.13
Development

No branches or pull requests
6 participants
@agnivade @myitcv @hyangah @bcmills @gopherbot @ryandbump