Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: get / mod: "module source tree too big" error #40780

Closed
ikawaha opened this issue Aug 14, 2020 · 2 comments
Closed

cmd/go: get / mod: "module source tree too big" error #40780

ikawaha opened this issue Aug 14, 2020 · 2 comments
Labels
FrozenDueToAge modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Unplanned

Comments

@ikawaha
Copy link

ikawaha commented Aug 14, 2020

What version of Go are you using (go version)?

go version go1.15 darwin/amd64

Does this issue reproduce with the latest release?

yes.

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE="auto"
GOARCH="amd64"
GOBIN="/Users/ikawaha/go/bin"
GOCACHE="/Users/ikawaha/Library/Caches/go-build"
GOENV="/Users/ikawaha/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/ikawaha/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/ikawaha/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/Users/ikawaha/sdk/go1.15"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/Users/ikawaha/sdk/go1.15/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/ikawaha/work/zzz/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/z5/_pm6q9c9263cv0cjx1b039h80000gn/T/go-build087421608=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

An error occurred when I tried to run the following program.

$ go run main.go
go: finding module for package github.com/ikawaha/kagome-dict-ipa-neologd
go: downloading github.com/ikawaha/kagome/v2 v2.0.4
go: downloading github.com/ikawaha/kagome-dict-ipa-neologd v0.0.1
main.go:6:2: create zip: module source tree too large (max size is 524288000 bytes)

main.go

package main

import (
        "fmt"

         dict "github.com/ikawaha/kagome-dict-ipa-neologd"
        "github.com/ikawaha/kagome/v2/tokenizer"
)

func main() {
        t, err := tokenizer.New(dict.Dict())
        if err != nil {
                panic(err)
        }
        // wakati
        fmt.Println("---wakati---")
        seg := t.Wakati("すもももももももものうち")
        fmt.Println(seg)
}

Kagome is a self-contained morphological analyzer that embeds dictionaries into binaries. I built a dictionary package for kagome using a named entity dictionary called Neologd, but got an error when I tried to get it.

The repository contains only the content of the dictionary as converted by go-bindata.

kagome-dict-ipa-neologd
$ du -h -d 1
600M	./internal
288M	./.git
 24K	./tool
889M	.

This error can be reproduced by go get github.com/ikawaha/kagome-dict-ipa-neologd or by running the above sample program.

What did you expect to see?

Successfully download the repository containing the large assets.

What did you see instead?

create zip: module source tree too large (max size is 524288000 bytes)
@jayconrod
Copy link
Contributor

Duplicate of #29210 and previously #25470, which raised the limit from 100MB to 500MB.

In practice, it's useful have this limit, since it prevents a lot of accidental abuse for proxy operators and for users. Most people who see this error have checked in large resources that aren't needed to build packages in their module, so the workaround is to move those resources into a directory with a go.mod file at the root (excluding them from the parent module) or to delete them from tagged commits.

This may be the first module we've seen that actually has more than 500MB of source code though. Still, the size limit is important, and other workarounds should be exhausted before increasing it.

  • Can the data be compressed or encoded in a more compact format within the source code?
  • Can this package be broken up into package in smaller modules?
  • Does the data need to be embedded into packages, or can it be fetched separately?

@jayconrod jayconrod added modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 14, 2020
@jayconrod jayconrod added this to the Unplanned milestone Aug 14, 2020
@ikawaha
Copy link
Author

ikawaha commented Aug 16, 2020

Thanks for the advice. I was able to get around the limitations by breaking the package into smaller modules.

@ikawaha ikawaha closed this as completed Aug 16, 2020
@loafoe loafoe mentioned this issue Aug 25, 2020
Closed
@golang golang locked and limited conversation to collaborators Aug 16, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
3 participants
@jayconrod @ikawaha @gopherbot