Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

sum.golang.org: returns 404 not found for an existing version of very large repo #56174

Open
tbruyelle opened this issue Oct 12, 2022 · 12 comments
Open

sum.golang.org: returns 404 not found for an existing version of very large repo #56174

tbruyelle opened this issue Oct 12, 2022 · 12 comments
Assignees
@suzmue
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. proxy.golang.org
Milestone
proxy.golang.org/...

Comments

@tbruyelle
Copy link
Contributor

tbruyelle commented Oct 12, 2022 •
edited

What version of Go are you using (go version)?

$ go version
go version go1.19.2 linux/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tom/.cache/go-build"
GOENV="/home/tom/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/tom/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/tom/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/lib/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/lib/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.19.2"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1236717261=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Update the ignite/cli dependency in a module to a newer version dffafa16f6fd91d9ef4462570038210c518d08dd

$ go get github.com/ignite/cli@dffafa16f6fd91d9ef4462570038210c518d08dd
go: downloading github.com/ignite/cli v0.24.1-0.20221012132336-dffafa16f6fd
go: github.com/ignite/cli@dffafa16f6fd91d9ef4462570038210c518d08dd: github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd: verifying module: github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd: reading https://sum.golang.org/lookup/github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd: 404 Not Found
	server response: not found:

The problem is reproducible without using a specific module :

  1. Fetch the pseudo-version
$ go list -m -json github.com/ignite/cli@dffafa16f6fd91d9ef4462570038210c518d08dd
{
	"Path": "github.com/ignite/cli",
	"Version": "v0.24.1-0.20221012132336-dffafa16f6fd",
	"Query": "dffafa16f6fd91d9ef4462570038210c518d08dd",
	"Time": "2022-10-12T13:23:36Z",
	"Origin": {
		"VCS": "git",
		"URL": "https://github.com/ignite/cli",
		"TagSum": "t1:yeejinhZSpi1V0OWB9y12hJfyz/MNKldv/v/ajQ87xE=",
		"Hash": "dffafa16f6fd91d9ef4462570038210c518d08dd"
	}
}
  1. Request sumdb with the pseudo-version
$ curl https://sum.golang.org/lookup/github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd
not found:

What did you expect to see?

sum.golang.org should return the checksum data instead of 404.

What did you see instead?

sum.golang.org returns a 404 not found error.
@hyangah
Copy link
Contributor

hyangah commented Oct 12, 2022 •
edited

Can you run the following command from a clean vanilla environment? (no special github permission, etc)

GOPROXY=direct GOPRIVATE=github.com/ignite/cli GOMODCACHE=/tmp/foo go mod download -x github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd

The command takes long time in my desktop (stuck during git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*' for a while. I guess sum.golang.org and proxy.golang.org have the same issue.

@hyangah hyangah added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 12, 2022
@tbruyelle
Copy link
Contributor Author

tbruyelle commented Oct 12, 2022 •
edited

GOPROXY=direct GOPRIVATE=github.com/ignite/cli GOMODCACHE=/tmp/foo go mod download -x github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd Output
mkdir -p /tmp/foo/cache/vcs # git3 https://github.com/ignite/cli
# lock /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0.lock
mkdir -p /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0 # git3 https://github.com/ignite/cli
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git init --bare
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git init --bare
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git remote add origin -- https://github.com/ignite/cli
0.005s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git remote add origin -- https://github.com/ignite/cli
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' dffafa16f6fd --
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' dffafa16f6fd --
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git ls-remote -q origin
0.612s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git ls-remote -q origin
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git tag -l
0.004s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git tag -l
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch -f --depth=1 origin refs/heads/feat/plugin-system-rebased:refs/heads/feat/plugin-system-rebased
10.996s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch -f --depth=1 origin refs/heads/feat/plugin-system-rebased:refs/heads/feat/plugin-system-rebased
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/heads/feat/plugin-system-rebased --
0.002s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' refs/heads/feat/plugin-system-rebased --
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
0.002s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' dffafa16f6fd91d9ef4462570038210c518d08dd --
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c log.showsignature=false log --no-decorate -n1 '--format=format:%H %ct %D' dffafa16f6fd91d9ef4462570038210c518d08dd --
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
162.735s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch -f origin 'refs/heads/*:refs/heads/*' 'refs/tags/*:refs/tags/*'
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch --unshallow -f origin
1.774s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git fetch --unshallow -f origin
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
0.004s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
0.002s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
0.003s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git merge-base --is-ancestor -- v0.24.0 dffafa16f6fd91d9ef4462570038210c518d08dd
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
0.002s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
0.002s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git cat-file blob dffafa16f6fd91d9ef4462570038210c518d08dd:go.mod
cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ dffafa16f6fd91d9ef4462570038210c518d08dd
9.448s # cd /tmp/foo/cache/vcs/34eb0ee23d95bbac94a613041a01736e397aac7df846b18804e1b27d755c43b0; git -c core.autocrlf=input -c core.eol=lf archive --format=zip --prefix=prefix/ dffafa16f6fd91d9ef4462570038210c518d08dd

I confirm that it's quite long on my machine too.

The repo has recently received a lot of commits with large binary files, do you think it's the cause of the 404 ? If sumdb times out because of that, I would have expected a different kind of error.

@tbruyelle
Copy link
Contributor Author

tbruyelle commented Oct 12, 2022 •
edited

Not that if I request the sumdb once its cache has expired, I have a slightly different error label (still 404 though) :

$ curl https://sum.golang.org/lookup/github.com/ignite/cli@v0.24.1-0.20221012132336-dffafa16f6fd
not found: fetch timed out

Looks like it's a time out issue indeed!

@hyangah hyangah added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Oct 13, 2022
@tbruyelle tbruyelle mentioned this issue Oct 17, 2022
Closed
@seankhliao seankhliao changed the title sum.golang.org returns 404 not found for an existing version sum.golang.org: returns 404 not found for an existing version Nov 19, 2022
@seankhliao seankhliao added this to the Unreleased milestone Nov 19, 2022
@topilski

This comment was marked as off-topic.

Sign in to view
@tbruyelle

This comment was marked as off-topic.

Sign in to view
@kung-foo

This comment was marked as off-topic.

Sign in to view
@seankhliao
Copy link
Member

#57185 is for what's happening now

@seankhliao seankhliao changed the title sum.golang.org: returns 404 not found for an existing version sum.golang.org: returns 404 not found for an existing version of very large repo Dec 9, 2022
@topilski

This comment was marked as off-topic.

Sign in to view
@zephyrtronium zephyrtronium mentioned this issue Mar 14, 2023
Closed
@zephyrtronium
Copy link
Contributor

My team ran into this yesterday with github.com/pdfcpu/pdfcpu@v0.4.1-0.20230312101853-a410fdc32533, but it took a while to figure out this was the issue. If there's no solution imminent, having some indication in the error message that the repo might be too large would save some time, especially where the error is currently just server response: not found: .

@zephyrtronium zephyrtronium mentioned this issue Mar 15, 2023
Open
@ikawaha ikawaha mentioned this issue Apr 5, 2023
Open
@zwinnerman-fleetdm zwinnerman-fleetdm mentioned this issue Sep 7, 2023
Closed
@bcmills
Copy link
Contributor

bcmills commented Nov 8, 2023

I've seen a few of these in the cmd/go regression tests on the Go builders, too; I've filed those as #64016.

@khanakia
Copy link

khanakia commented Jan 6, 2024

I am getting not found error but if you check the repo already exist here https://github.com/ubgo/gofm/releases/tag/ginserver%2Fv0.0.9

tst go get github.com/ubgo/gofm/ginserver@v0.0.9                             
go: downloading github.com/ubgo/gofm/ginserver v0.0.9
go: github.com/ubgo/gofm/ginserver@v0.0.9: verifying module: github.com/ubgo/gofm/ginserver@v0.0.9: reading https://sum.golang.org/lookup/github.com/ubgo/gofm/ginserver@v0.0.9: 404 Not Found
	server response:
	not found: github.com/ubgo/gofm/ginserver@v0.0.9: invalid version: git ls-remote -q origin in /tmp/gopath/pkg/mod/cache/vcs/7f2330e9251df817986bc960f2658c75dcadbfc629e4df795b6b857268f84e8f: exit status 128:
		fatal: could not read Username for 'https://github.com': terminal prompts disabled
	Confirm the import path was entered correctly.
	If this is a private repository, see https://golang.org/doc/faq#git_https for additional information.

@bitoku bitoku mentioned this issue Feb 14, 2024
Open
@alexaandru
Copy link

alexaandru commented Apr 12, 2024 •
edited

I believe I'm running into the same issue with https://github.com/alexaandru/go-sitter-forest .

Even though https://github.com/alexaandru/go-sitter-forest/tree/bash/v1.5.11 clearly exists (for about 30 minutes now), go get github.com/alexaandru/go-sitter-forest/bash@latest and go get github.com/alexaandru/go-sitter-forest/bash@v1.5.11 both fail with server response: not found: github.com/alexaandru/go-sitter-forest/bash@v1.5.11: invalid version: unknown revision bash/v1.5.11.

The repo is indeed very large (1.3GB) but each folder in there is an individual Go module. Bash folder alone is only 10MB and the .git folder alone is ~130MB.

Update: 5 minutes later, it worked! :-)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@suzmue suzmue

Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. proxy.golang.org
Projects
None yet
Milestone
proxy.golang.org/backlog
Development

No branches or pull requests