proxy.golang.org returning 410 for k8s.io/client-go@v12.0.0+incompatible #33558

davecheney · 2019-08-09T01:00:25Z

proxy.golang.org has stoped returning the correct result for k8s.io/client-go v 12.0.0

(~/devel/tttt) % cat go.mod 
module whatever

go 1.12

require k8s.io/client-go v12.0.0+incompatible // indirect
(~/devel/tttt) % env GOPATH=$(mktemp -d) GOPROXY=https://proxy.golang.org go1.12 get -x k8s.io/client-go@v12.0.0+incompatible
go: finding k8s.io/client-go v12.0.0+incompatible
go: k8s.io/client-go@v12.0.0+incompatible: unexpected status (https://proxy.golang.org/k8s.io/client-go/@v/v12.0.0+incompatible.info): 410 Gone
go: error loading module requirements

Note: the env GOPATH... is to avoid having to sudo rm -rf my $HOME/pkg/mod directory.

This was working on Friday and has broken our CI.

The text was updated successfully, but these errors were encountered:

thepudds · 2019-08-09T01:06:03Z

You have likely already seen this, but some chance related to #33513 (comment)

bcmills · 2019-08-09T01:23:34Z

Note the error text returned by that URL:

not found: k8s.io/client-go@v12.0.0+incompatible: invalid version: +incompatible suffix not allowed: module contains a go.mod file, so semantic import versioning is required

See https://tip.golang.org/doc/go1.13#version-validation for more detail and advice on fixes and workarounds.

bcmills · 2019-08-09T01:24:43Z

See also #32695.

bcmills · 2019-08-09T01:26:35Z

Duplicate of #32805

thepudds · 2019-08-09T01:34:48Z

FWIW, a bit more detail.

The v12.0.0 go.mod file for k8s.io/client-go appears to be missing the required /v12 at the end of the module line:

https://github.com/kubernetes/client-go/blob/v12.0.0/go.mod#L3

module k8s.io/client-go

From the cmd/go documentation

In semantic versioning, changing the major version number indicates a lack of backwards compatibility with earlier versions. To preserve import compatibility, the go command requires that modules with major version v2 or later use a module path with that major version as the final element. For example, version v2.0.0 of example.com/m must instead use module path example.com/m/v2, and packages in that module would use that path as their import path prefix, as in example.com/m/v2/sub/pkg. Including the major version number in the module path and import paths in this way is called "semantic import versioning".

CC @liggitt @nikhita

thepudds · 2019-08-09T01:43:53Z

Some additional context is that proxy.golang.org was recently updated to use 1.13beta1, which included "invalidating some old module versions that are currently being served from proxy.golang.org but are no longer valid in 1.13".

Some more details from the one of the announcements on golang-dev:

As an update, related to https://golang.org/issue/32805 and https://golang.org/issue/32879#issuecomment-507861612:
Go 1.13 includes some changes to the go command which tighten up the validation of module versions. Now that we have upgraded proxy.golang.org to use the 1.13beta1 version of the go command, we will be invalidating old module versions that are currently being served from proxy.golang.org but are no longer valid in 1.13. We don't expect any major issues from users since the request count is low for each of the module versions that will be invalidated, but please file issues against the issue tracker, prefixed "proxy.golang.org", if you encounter any problems. Thanks to everyone who has filed issues so far and given us feedback!

davecheney · 2019-08-09T01:46:55Z

@bcmills i'm sorry to reopen this but this has broken our Go 1.12 builds, we're not in a position to upgrade to 1.13beta1. We need a solution that restores functionality for Go 1.12.

bcmills · 2019-08-09T01:51:37Z

@davecheney, a replace directive (as described in the 1.13 release notes) should also work in 1.12. Please give it a try.

thepudds · 2019-08-09T01:59:12Z

Two other potential options when using the Go 1.12 go command:

Use an alternate proxy. For example, this seemed to work just now, perhaps because goproxy.io is not yet running Go 1.13:

GOPROXY=https://goproxy.io go get -d k8s.io/client-go@v12.0.0

Use direct. This also seemed to work just now:

GOPROXY=direct go get -d k8s.io/client-go@v12.0.0

davecheney · 2019-08-09T02:10:28Z

@bcmills this worked last week. Our builds built last week using Go 1.12. They worked on Monday. Something has broken in proxy.golang.org that means it no longer works as it did just a few days ago.

It's unreasonable to ask users to change their go.mod configuration because proxy.golang.org can no longer serve valid artefacts that it served a few days ago.

davecheney · 2019-08-09T02:12:32Z

Further,

We don't expect any major issues from users since the request count is low for each of the module versions that will be invalidated, but please file issues against the issue tracker, prefixed "proxy.golang.org", if you encounter any problems.

Clearly this is not true, v12 of client-go is a very popular dependency and now it is inaccessible for Go 1.12 users using proxy.golang.org.

thepudds · 2019-08-09T02:21:55Z

One other possible solution is I think 1.12 can be forced to use a pseudo-version with a commit hash corresponding to the v12.0.0 tag of k8s.io/client-go.

In particular, this works:

GOPROXY=https://proxy.golang.org go get -d k8s.io/client-go@78d2af7

with this result recorded in go.mod:

require k8s.io/client-go v0.0.0-20190620085101-78d2af792bab

78d2af792bab is the commit hash for v12.0.0 of client-go.

An alternative path to a similar result:

echo 'replace k8s.io/client-go => k8s.io/client-go 78d2af7' >> go.mod
go list -m all

which then results in a go.mod with:

replace k8s.io/client-go => k8s.io/client-go v0.0.0-20190620085101-78d2af792bab

bcmills · 2019-08-09T02:22:55Z

It's unreasonable to ask users to change their go.mod configuration because proxy.golang.org can no longer serve valid artefacts that it served a few days ago.

The v12.0.0 artifact is not valid and never was — as far as I'm aware the requirement for semantic import paths has been clearly documented since Go 1.11.

The fact that v12.0.0+incompatible could be fetched in certain circumstances by playing games with manually-added suffixes was a bug, plain and simple. The fact that that bug was temporarily exacerbated by the proxy's /list endpoint is unfortunate, but now that's been fixed too.

bcmills · 2019-08-09T02:37:37Z

See also kubernetes/client-go#631.

davecheney · 2019-08-09T02:39:11Z

The fact that v12.0.0+incompatible could be fetched in certain circumstances by playing games with manually-added suffixes was a bug, plain and simple.

It may be a bug, but the bug exists in Go 1.12, the version that produced that line. Saying it's a bug is fine, but breaking projects that were happily working up to a few days ago is pretty unfriendly behaviour.

davecheney · 2019-08-09T02:47:29Z

@bcmills given that go.mod files with lines ending in +incompatible can no longer be served via proxies, should the go command reject them completely when operating in proxy mode?

bcmills · 2019-08-09T02:48:27Z

The bug in Go 1.12 required manual exploitation by users (adding an explicit +incompatible suffix). Otherwise, they would get either an error or a version rewritten to the valid equivalent (v0.0.0-20190620085101-78d2af792bab).

The only users who would have picked it up without manual intervention would have picked it up from a cache or proxy entry introduced by some other module that manually exploited the bug.

Moreover, there is a simple workaround available: replace the invalid version with its valid equivalent.

bcmills · 2019-08-09T02:52:21Z

Adding the same validation to the proxy path is #32955, but that's a harder problem in general because the protocol erases version control history. (It's possible to do for invalid +incompatible versions, but not for invalid pseudo-versions because the proxy does not supply compete version-control history.)

davecheney · 2019-08-09T02:57:40Z

Thanks for clarifying. If +incompatible is a bug or represents incorrect manual editing, should the go tool error out when it finds those lines? If the hypothesis is the impact of the proxy.golang.org change is small, then there should only be a small number of +incompatible lines in the wild, so rejecting them quickly so module authors can fix the problem seems line a nice thing to do. Would you agree?

bcmills · 2019-08-09T03:05:04Z

+incompatible is only a bug when used with a version that either has an explicit go.mod file or is not actually incompatible with the given module path.

Those really are rare: we ran several analyses to verify that (see #27173 (comment), and the folks running proxy.golang.org did their own internal analysis before removing the invalid versions).

davecheney · 2019-08-09T03:07:42Z

I don't think they are as rare as you believe.

Here are two that I found in the go.mod file for client-go

https://github.com/kubernetes/client-go/blob/master/go.mod#L8

https://github.com/kubernetes/client-go/blob/master/go.mod#L21

bcmills · 2019-08-09T03:11:54Z

@davecheney, please read my replies above more carefully:

+incompatible is only a bug when used with a version that either has an explicit go.mod file or is not actually incompatible with the given module path.

The two versions you note in your comment above both lack explicit go.mod files, and are therefore correct uses of the +incompatible annotation.

thepudds · 2019-08-09T03:13:32Z

One quick comment: +incompatible itself is not a problem, nor is it rare.

Any module can import a v2+ package that has not opted into modules, and if the imported v2+ package has a valid semver tag, the imported package will be recorded with an +incompatible suffix in the importing module's go.mod file. That is fine and expected behavior, and an important transitional mechanism that helps modules consume non-module v2+ packages.

Using your first example:
github.com/Azure/go-autorest v11.1.2+incompatible

That is fine, because github.com/Azure/go-autorest@v11.1.2 itself does not have a go.mod file.

The main problem being discussed in this thread is when a v2+ package attempts to adopt modules, but does not follow the requirement listed in the doc quote in
#33558 (comment).

In particular:

The v12.0.0 go.mod file for k8s.io/client-go appears to be missing the required /v12 at the end of the module line:

https://github.com/kubernetes/client-go/blob/v12.0.0/go.mod#L3

module k8s.io/client-go

That module line in the client-go go.mod file should read module k8s.io/client-go/v12 (note the trailing /v12) to conform with that documentation quote in #33558 (comment).

davecheney · 2019-08-09T03:19:34Z

ahh, ok. Thanks for explaining. So the rules for +incompatible are:

if the module does not have a go.mod file and has a sevmer tag above v1, then it is ok to list it as +incompatible. (I'm assuming the go tool, or the proxy, strips off the +incompatable at some point)
If the module does have a go.mod file and has a semver tag above v1 but does not conform to SIV, then the module cannot be referred to as +incompatible, the importer must fall back to the v0 pseudo version.

bcmills · 2019-08-09T03:20:44Z

Correct.

bcmills · 2019-08-09T15:01:29Z

As another data-point: a search for k8s.io/client-go at v12.0.0+incompatible turns up only one affected go.mod file (in github.com/kubernetes/ingress-nginx, with the replace workaround applied in kubernetes/ingress-nginx@f93bcfb), but a search for v0.0.0-20190620085101-78d2af792bab (the equivalent valid version) finds many occurrences in go.mod files throughout the ecosystem.

That suggests that (consistent with our hypothesis) most users were not affected by the buggy version served by proxy.golang.org. And that's to be expected, since Go 1.12 had to be configured explicitly in order to use proxy.golang.org.

I'm sorry that you were one of the unlucky few, but you do seem to be an outlier here.

bcmills · 2019-08-09T15:59:19Z

CC @katiehockman @hyangah @heschik

heschi · 2019-08-09T18:05:33Z

I agree with Bryan on all fronts. I double-checked the logs and v12.0.0+incompatible has relatively few downloads; I don't think we made a big mistake removing it. Much better to match behavior with 1.13, where it will be the default, even if it causes some trouble for 1.12 users. Hopefully there won't be another upheaval like this in 1.14 -- I can't imagine us doing anything like this once the service is out of beta.

Sorry for the trouble, but this is why it's still in beta. We hope to fully launch very soon.

ankushchadha · 2019-08-09T18:43:10Z

k8s.io/client-go@v12.0.0+incompatible is referred by 130 go module versions and what it means is that their builds will be broken unless they set the GOPROXY to https://gocenter.io. We will continue to serve this version. This specific version was downloaded a lot of times (in thousands) from GoCenter.
More details here - https://search.gocenter.io/k8s.io~2Fclient-go/info?version=v12.0.0%2Bincompatible

bcmills · 2019-08-09T18:58:25Z

@ankushchadha, as far as I can tell, most (perhaps all?) of those 130 modules only appear to have such a dependency due to a bug in GoCenter itself, not due to anything explicitly in their go.mod files or transitive dependencies.

Arbitrarily checking the “Mod File” tab for the first eight entries listed on the “Used By” tab of https://search.gocenter.io/k8s.io~2Fclient-go/info?version=v12.0.0%2Bincompatible, I see the following note on five of them:

It seems this project has not adopted Go modules yet. Here's a suggested go.mod file for the latest version.

The other three modules I sampled from the “Used By” list show entirely different versions of k8s.io/client-go altogether!

github.com/concourse/concourse lists k8s.io/client-go v2.0.0-alpha.0.0.20171101191150-72e1c2a1ef30+incompatible.
github.com/pachyderm/pachyderm lists k8s.io/client-go v0.0.0-20181213230135-6924ba6dfc02.
github.com/kubeflow/kubeflow shows a “Mod File” tab containing only a module directive and lists no dependencies whatsoever, and the underlying version as shown in GitHub does not even have a go.mod file!
- That repository does, however, contain several nested modules that require k8s.io/client-go at various different versions, none of which are v12.0.0+incompatible.

So I'm led to believe that what you are seeing is a significant bug in GoCenter's version indexing rather than organic usage of the invalid version.

ankushchadha · 2019-08-09T22:24:01Z

That's the current UX if the dependents have not adopted Go modules. However, there are 58 references for "k8s.io/client-go v12.0.0+incompatible" on GitHub (https://github.com/search?p=2&q=%22k8s.io%2Fclient-go+v12.0.0%2Bincompatible%22&type=Code)

I don't know how many private projects refer to this version of k8s.io/client-go.

In addition, there are 19857 modules above v1 with +incompatible suffix in GoCenter. We don't know how many of them have go.mod file but we imagine that the impact of this change is going to be huge since these modules along with dependents may not be usable anymore.

GoCenter will continue to honor the immutability and availability requirement and therefore continue to serve these module versions to achieve reproducible builds for the end-users.

The k8s.io/client-go v12 go.mod file has an error. See golang/go#33558 Signed-off-by: Bryan Boreham <bryan@weave.works>

Upgrade client-go 11.0.0→11.0.0+patches, consul 1.4.4→1.5.0 A while back, I tried upgrading client-go to v12 (kube 1.15) in teleproxy.git, mostly just as an exercise to see how difficult it would be, not because I actually wanted to switch to v12. I failed to do it, and ended up in dependency hell. I came to the conclusion that trying to manage client-go's dependencies *and* your own dependencies at the same time was a nightmare, and that it would be a good idea to have a separate library just to isolate the dependencies of client-go. I made that library! https://github.com/datawire/libk8s. Currently, it does nothing but anonymous-import every part of client-go, and properly declare all of the versions of dependencies. > Spoilers: There are probably 2 more reasons that extra-contributed > to my trouble: > - client-go v12 claims to be a valid Go module, but isn't. The fact > that the Go 1.12 compiler accepts it is a bug in the compiler (the > Go 1.13 compiler does not) > <golang/go#33558 (comment)>. > I was using Go 1.12.7, but I figure this didn't help > - It got really confused about consul. If you try to resolve a > package in `github.com/hashicorp/consul/api/...`, it discovers > "yeah, there's a `github.com/hashicorp/consul/api` module (that is > a sub-module of `github.com/hashicorp/consul`)!" But it didn't > become a sub-module until consul v1.5.0; we're using v1.4.4, in > which it wasn't split off yet. The whole thing confuses `go mod > tidy`. Also, we'd been using the `v11.0.0` tag of `client-go`. Did you know that there's a `release-11.0` branch that gets bug-fixes and security fixes!? OK, so this PR is about equal parts of several things: - Try out using https://github.com/datawire/libk8s to manage the dependency on client-go. - Try out upgrading client-go (still v11, but with a few commits from the `release-11.0` branch) - Figure out why the hell `go mod tidy` was breaking the build (answer: consul) - I tried upgrading the version of the teleproxy library that APro uses, and got some weird error. Hopefully this fixes it.

…v1.15.4 of k8s api/apimachinery in sync with other voltha components Had to use pseudo-version corresponding to v12.0.0 of k8s client-go because golang proxy is no longer serving the modules not complying to Semantic Import Versioning rules including client-go v12.0.0. Refer to kubernetes/client-go#631 and golang/go#33558 Change-Id: I2e558bab7f0702f230761319eb5392a7d0532ea3

mattn · 2019-11-18T06:58:33Z

Sorry if my comment is not related on this issue.

mattn/go-sqlite3#755

Last year, in go-sqlite3, topic branch v2.0.0 was created to ready to make new release. Unfortunately, v2.0.0 was not merged. mattn/go-sqlite3#611

So I removed the topic branch with git push origin :v2.0.0 but proxy.golang.org return https://proxy.golang.org/github.com/mattn/go-sqlite3/@v/list

v1.9.0
v1.3.0
v1.7.0
v1.8.0
v1.6.0
v1.0.0
v1.12.0
v1.5.0
v2.0.0+incompatible
v1.10.0
v1.11.0
v1.4.0
v1.2.0
v1.1.0

Is there any other way to solve this problem without releasing v2.0.1 of go-sqlite3 ?

Note) we did not make git tag v2.0.0

bcmills · 2019-11-18T15:22:42Z

@mattn, see #34189. As of Go 1.14, if the latest compatible version has a go.mod file, the go command will prefer the compatible version over any potential +incompatible one. So it may suffice to add a go.mod file at the root of the github.com/mattn/go-sqlite3 repo.

pierrre · 2019-11-18T19:51:02Z

What is the short-term solution ?
Go 1.14 is not yet released.
Is it possible to purge the proxy.golang.org cache ?

ergoz · 2019-11-21T16:07:00Z

this issue broke 20+ my packages

grosser · 2020-05-16T18:43:40Z

can we please get a real/automatic fix for this ?
... super annoying to have to google for a bug and then dig through this big issue just to find out that you have to copy-paste some hack into your go.mod :(

bcmills · 2020-05-19T02:45:14Z

@grosser, see kubernetes/client-go#749.

#24031 on the Go side may provide another option, in that it would allow the Kubernetes maintainers to retract all of the releases higher than v0.

bcmills marked this as a duplicate of #32805 Aug 9, 2019

bcmills closed this as completed Aug 9, 2019

davecheney reopened this Aug 9, 2019

bcmills added modules WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Aug 9, 2019

bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 9, 2019

bcmills added this to the Go1.13 milestone Aug 9, 2019

bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 9, 2019

bcmills mentioned this issue Aug 9, 2019

Update go dependencies kubernetes/ingress-nginx#4322

Merged

heschi closed this as completed Aug 9, 2019

LukeShu mentioned this issue Sep 2, 2019

Upgrade client-go 11.0.0→11.0.0+patches, consul 1.4.4→1.5.0 datawire/teleproxy#172

Merged

bboreham added a commit to cortexproject/cortex that referenced this issue Sep 9, 2019

Override reference that causes an error from Go proxy

5b840f8

The k8s.io/client-go v12 go.mod file has an error. See golang/go#33558 Signed-off-by: Bryan Boreham <bryan@weave.works>

bboreham mentioned this issue Sep 9, 2019

Override reference that causes an error from Go proxy cortexproject/cortex#1660

Merged

mattn mentioned this issue Nov 18, 2019

cannot load github.com/mattn/go-sqlite3: module github.com/mattn/go-sqlite3@latest (v2.0.0+incompatible) found, but does not contain package github.com/mattn/go-sqlite3 mattn/go-sqlite3#755

Closed

This was referenced Nov 18, 2019

proxy.golang.org: accidentally publishing a major version interplays poorly with Go modules #34189

Closed

cmd/go: do not treat branches with semantic-version names as releases #35671

Closed

jsimonetti mentioned this issue Dec 4, 2019

Fix incompatible go proxy format networkop/meshnet-cni#15

Merged

harsh-px mentioned this issue Jan 23, 2020

[PTX-1641] Added AKS node driver portworx/torpedo#389

Closed

marianrh mentioned this issue Aug 25, 2020

When using v1.14.1, go get -u ./... wants to fetch v2.0.6+incompatible and fails mattn/go-sqlite3#844

Closed

golang locked and limited conversation to collaborators May 19, 2021

gopherbot added the FrozenDueToAge label May 19, 2021

proxy.golang.org returning 410 for k8s.io/client-go@v12.0.0+incompatible #33558

proxy.golang.org returning 410 for k8s.io/client-go@v12.0.0+incompatible #33558

Comments

davecheney commented Aug 9, 2019 • edited

thepudds commented Aug 9, 2019

bcmills commented Aug 9, 2019

bcmills commented Aug 9, 2019

bcmills commented Aug 9, 2019

thepudds commented Aug 9, 2019

thepudds commented Aug 9, 2019 • edited

davecheney commented Aug 9, 2019

bcmills commented Aug 9, 2019

thepudds commented Aug 9, 2019

davecheney commented Aug 9, 2019

davecheney commented Aug 9, 2019

thepudds commented Aug 9, 2019 • edited

bcmills commented Aug 9, 2019 • edited

bcmills commented Aug 9, 2019

davecheney commented Aug 9, 2019 • edited

davecheney commented Aug 9, 2019

bcmills commented Aug 9, 2019 • edited

bcmills commented Aug 9, 2019

davecheney commented Aug 9, 2019

bcmills commented Aug 9, 2019

davecheney commented Aug 9, 2019

bcmills commented Aug 9, 2019 • edited

thepudds commented Aug 9, 2019 • edited

davecheney commented Aug 9, 2019 • edited

bcmills commented Aug 9, 2019

bcmills commented Aug 9, 2019 • edited

bcmills commented Aug 9, 2019

heschi commented Aug 9, 2019

ankushchadha commented Aug 9, 2019

bcmills commented Aug 9, 2019 • edited

ankushchadha commented Aug 9, 2019 • edited

mattn commented Nov 18, 2019 • edited

bcmills commented Nov 18, 2019

pierrre commented Nov 18, 2019

ergoz commented Nov 21, 2019

grosser commented May 16, 2020

bcmills commented May 19, 2020

davecheney commented Aug 9, 2019 •

edited

thepudds commented Aug 9, 2019 •

edited

thepudds commented Aug 9, 2019 •

edited

bcmills commented Aug 9, 2019 •

edited

davecheney commented Aug 9, 2019 •

edited

bcmills commented Aug 9, 2019 •

edited

bcmills commented Aug 9, 2019 •

edited

thepudds commented Aug 9, 2019 •

edited

davecheney commented Aug 9, 2019 •

edited

bcmills commented Aug 9, 2019 •

edited

bcmills commented Aug 9, 2019 •

edited

ankushchadha commented Aug 9, 2019 •

edited

mattn commented Nov 18, 2019 •

edited