Skip to content

x/vgo: how to deal with kubernetes client-go v6.0.0 #24032

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
folago opened this issue Feb 22, 2018 · 19 comments
Closed

x/vgo: how to deal with kubernetes client-go v6.0.0 #24032

folago opened this issue Feb 22, 2018 · 19 comments
Labels
FrozenDueToAge
Milestone
vgo

Comments

@folago
Copy link

folago commented Feb 22, 2018 •
edited
Loading

What version of Go are you using (go version)?

$ go version
go version go1.10 darwin/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

$ go env
GOARCH="amd64"
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
...

What did you do?

vgo build of a binary command in a subdirectory with go.mod and dep lock files in the root directory.
nlb is the root directory, the structure is the following:

  • nlb
    • Gopkg.*
    • go.mod
    • vendor
    • cmd
      • nlbcontroller
        • main.go run_vgo_here

What did you expect to see?

successful build

What did you see instead?

$ vgo build
vgo: resolving import "k8s.io/api/core/v1"
vgo: finding k8s.io/api (latest)
vgo: adding k8s.io/api v0.0.0-20180216210113-b378c47b2dcb
vgo: resolving import "k8s.io/apimachinery/pkg/apis/meta/v1"
vgo: finding k8s.io/apimachinery (latest)
vgo: adding k8s.io/apimachinery v0.0.0-20180216125745-cced8e64b6ca
vgo: resolving import "k8s.io/client-go/kubernetes"
vgo: finding k8s.io/client-go (latest)
vgo: adding k8s.io/client-go v1.5.1
vgo: finding k8s.io/client-go v1.5.1
vgo: finding k8s.io/apimachinery v0.0.0-20180216125745-cced8e64b6ca
vgo: finding gopkg.in/yaml.v2 v0.0.0-20170721113624-670d4cfef054
vgo: finding gopkg.in/inf.v0 v0.0.0-20150911125757-3887ee99ecf0
vgo: finding golang.org/x/text v0.0.0-20170810154203-b19bf474d317
vgo: finding golang.org/x/net v0.0.0-20170809000501-1c05540f6879
vgo: finding github.com/stretchr/testify v0.0.0-20170601210322-f6abca593680
vgo: finding github.com/stretchr/objx v0.0.0-20140526180921-cbeaeb16a013
vgo: finding github.com/davecgh/go-spew v0.0.0-20161028175848-04cdfd42973b
vgo: finding github.com/spf13/pflag v0.0.0-20171106142849-4c012f6dcd95
vgo: finding github.com/pmezard/go-difflib v0.0.0-20151028094244-d8ed2627bdf0
vgo: finding github.com/pborman/uuid v0.0.0-20150603214016-ca53cad383ca
vgo: finding github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f
vgo: finding github.com/json-iterator/go v0.0.0-20171212105241-13f86432b882
vgo: finding github.com/stretchr/testify v1.1.4
vgo: finding github.com/davecgh/go-spew v0.0.0-20160907170601-6d212800a42e
vgo: finding github.com/pmezard/go-difflib v1.0.0
vgo: finding github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf
vgo: finding github.com/davecgh/go-spew v1.1.0
vgo: finding github.com/hashicorp/golang-lru v0.0.0-20160207214719-a0d98a5f2880
vgo: finding github.com/googleapis/gnostic v0.0.0-20170729233727-0c5108395e2d
vgo: finding github.com/google/gofuzz v0.0.0-20161122191042-44d81051d367
vgo: finding github.com/golang/protobuf v0.0.0-20171021043952-1643683e1b54
vgo: finding github.com/golang/groupcache v0.0.0-20160516000752-02826c3e7903
vgo: finding github.com/golang/glog v0.0.0-20141105023935-44145f04b68c
vgo: finding github.com/gogo/protobuf v0.0.0-20170330071051-c0656edd0d9e
vgo: finding github.com/ghodss/yaml v0.0.0-20150909031657-73d445a93680
vgo: finding github.com/evanphx/json-patch v0.0.0-20170719203123-944e07253867
vgo: finding github.com/elazarl/goproxy v0.0.0-20170405201442-c4fc26588b6e
vgo: finding github.com/docker/spdystream v0.0.0-20160310174837-449fdfce4d96
vgo: finding github.com/davecgh/go-spew v0.0.0-20170626231645-782f4967f2dc
vgo: finding k8s.io/api v0.0.0-20180216210113-b378c47b2dcb
vgo: downloading k8s.io/api v0.0.0-20180216210113-b378c47b2dcb
vgo: downloading github.com/gogo/protobuf v0.0.0-20170330071051-c0656edd0d9e
vgo: downloading k8s.io/apimachinery v0.0.0-20180216125745-cced8e64b6ca
vgo: downloading github.com/spf13/pflag v0.0.0-20171106142849-4c012f6dcd95
vgo: downloading gopkg.in/inf.v0 v0.0.0-20150911125757-3887ee99ecf0
vgo: downloading github.com/google/gofuzz v0.0.0-20170612174753-24818f796faf
vgo: downloading golang.org/x/net v0.0.0-20170809000501-1c05540f6879
vgo: downloading golang.org/x/text v0.0.0-20170810154203-b19bf474d317
vgo: downloading k8s.io/client-go v1.5.1
vgo: import "github.com/folago/nlb/cmd/nlbcontroller" ->
import "k8s.io/client-go/kubernetes" [/Users/gta/go/src/v/k8s.io/client-go@v1.5.1/kubernetes]: open /Users/gta/go/src/v/k8s.io/client-go@v1.5.1/kubernetes: no such file or directory
vgo: import "github.com/folago/nlb/cmd/nlbcontroller" ->
import "k8s.io/client-go/rest" [/Users/gta/go/src/v/k8s.io/client-go@v1.5.1/rest]: open /Users/gta/go/src/v/k8s.io/client-go@v1.5.1/rest: no such file or directory
vgo: import "github.com/folago/nlb/cmd/nlbcontroller" ->
import "k8s.io/client-go/tools/cache" [/Users/gta/go/src/v/k8s.io/client-go@v1.5.1/tools/cache]: open /Users/gta/go/src/v/k8s.io/client-go@v1.5.1/tools/cache: no such file or directory
vgo: import "github.com/folago/nlb/cmd/nlbcontroller" ->
import "k8s.io/client-go/util/workqueue" [/Users/gta/go/src/v/k8s.io/client-go@v1.5.1/util/workqueue]: open /Users/gta/go/src/v/k8s.io/client-go@v1.5.1/util/workqueue: no such file or directory

$ ll /Users/gta/go/src/v/k8s.io/client-go@v1.5.1/
total 56
drwxr-xr-x 13 gta staff 416B Feb 21 17:20 1.5/
-rw-r--r-- 1 gta staff 11K Feb 21 17:20 LICENSE
-rw-r--r-- 1 gta staff 2.7K Feb 21 17:20 README.md
-rw-r--r-- 1 gta staff 7.2K Feb 21 17:20 copy.sh
drwxr-xr-x 4 gta staff 128B Feb 21 17:20 examples/
-rw-r--r-- 1 gta staff 2.2K Feb 21 17:20 godeps-json-updater.go

The expected directory structure is the following:
$ll ../../vendor/k8s.io/client-go/
total 24
-rw-r--r-- 1 gta staff 11K Feb 20 10:44 LICENSE
drwxr-xr-x 7 gta staff 224B Feb 20 10:44 discovery/
drwxr-xr-x 8 gta staff 256B Feb 20 10:44 kubernetes/
drwxr-xr-x 3 gta staff 96B Feb 20 10:44 pkg/
drwxr-xr-x 14 gta staff 448B Feb 20 10:44 rest/
drwxr-xr-x 7 gta staff 224B Feb 20 10:44 tools/
drwxr-xr-x 8 gta staff 256B Feb 20 10:44 transport/
drwxr-xr-x 7 gta staff 224B Feb 20 10:44 util/
@gopherbot gopherbot added this to the vgo milestone Feb 22, 2018
@kardianos
Copy link
Contributor

It looks like the k8s.io/client-go points to github releases. The v1.5.1 release can be found here: https://github.com/kubernetes/client-go/releases/tag/v1.5.1
When I go to that tag in the src or unzip that file, I'm not seeing the expected sub-packages in it. I see a top level 1.5 folder and an examples folder.

I suspect some dependency has a dep spec file or similar that specifies a min version of 1.5.1, but in reality you need something much newer. I suspect vgo is acting correctly in this case and we need to work on the ecosystem to correctly specify dependencies.

@danderson
Copy link
Contributor

The k8s client library is notoriously incredibly hard to use or install.

The readme recommends using the v6.0.0 tag. Sadly, this is incompatible with semantic import versioning, and vgo rejects an attempt to use v6.0.0 because there is no "v6" in the import path.

I suspect the k8s.io custom domain may be further confusing vgo into not using whatever specialized logic it might have for Github.

Given this, I'm not sure how to get vgo to work with k8s.io/client-go. AFAICT, if vgo were to bypass its semantic import versioning check, everything else would correctly fall into place. @rsc, any ideas/pointers?

@danderson
Copy link
Contributor

Specifically: since the import path does not imply any major version, I believe vgo is searching for a 1.x.x tag, and lands on 1.5.2. By Kubernetes standards this is ancient (3 years old at least, I believe), so I'm not particularly surprised that it doesn't build or have usable dependency data.

@kardianos
Copy link
Contributor

You are correct. Kube client would need to commit a go.mod file with the correct v6 module path and users would need to update to use it.

@danderson
Copy link
Contributor

Sadly, that makes it very likely that anything touching Kubernetes will be unable to prototype vgo. It seems pretty unlikely to me that the client library and its related dependencies will update during the prototype phase (or ever, if I'm feeling perfectly candid :( )

@kardianos
Copy link
Contributor

kardianos commented Feb 23, 2018 via email

@danderson
Copy link
Contributor

Hmm, I'm confused. I thought vgo required that the v6.x code lives in a v6/ subdirectory. That's a significantly larger change than just adding a module file.

@kardianos
Copy link
Contributor

kardianos commented Feb 23, 2018 via email

@danderson
Copy link
Contributor

Got it, thanks. So, the minimum required change is:

  • Add go.mod to github.com/kubernetes/client-go, declaring the module to be k8s.io/client-go/v6.
  • Update all of client-go's internal imports to insert /v6/, so that vgo doesn't end up trying to use v6 and v1 at the same time.
  • Repeat for other k8s repositories like apimachinery, which are likely using the same pattern.

I'll fork the modules and see what I can get working.

@kardianos
Copy link
Contributor

Exactly.

@danderson
Copy link
Contributor

Okay, done. If anyone else finds this bug trying to make the Kubernetes client lib work (and assuming you're using client-go v6.0.0), add the following to your go.mod:

require "k8s.io/client-go/v6" v6.0.0
replace "k8s.io/client-go/v6" v6.0.0 => "github.com/danderson/client-go" v0.0.0-vgo-prototype-compat

No guarantees at all that this produces correct binaries, but it seems to successfully compile mine, at least.

Now, the question is: how can we upstream danderson/client-go@7d84811 to the client library without breaking all current users? AFAICT we'll have to keep using a replace'd library until the final form of this is upstreamed into go, otherwise there's no way of making this repository work correctly for both vgo and legacy go.

@kardianos
Copy link
Contributor

What if you added a v6 symlink to the client repo as well, then see if it compiles with both vgo and go with the /v6 amended import paths? That might break on windows though.

@yiyus
Copy link

yiyus commented Feb 24, 2018

@danderson I have not tested it, but couldn't you just use replace to avoid having to modify import paths? The go.mod file in client-go would take the form:

module "k8s.io/client-go/v6"
replace "k8s.io/client-go/" v0.0.0 => "k8s.io/client-go/v6" v6.0.0

Then, both users of go get and users of vgo would be able to use the module.

If this is not possible, I think it should be, and even be the default for every module. It would make easier to transition to newer versions internally, because only the go.mod file would need to change. The import paths inside the module would always be unversioned ones, but old versions would still be accessible from newer ones and vgo users would have to use versioned paths (the transition may again be facilitated by the use of replace).

In a vgo only world, I think it would make sense to use relative import paths for the packages inside a module and remove import comments.

@rsc rsc changed the title x/vgo: build fails to download subdirectories x/vgo: how to deal with kubernetes client-go v6.0.0 Apr 2, 2018
@rsc
Copy link
Contributor

rsc commented Apr 2, 2018

I retitled this to be about kubernetes client-go, although the problem is more general. The issue is that the import said client-go, so vgo took the latest v1, which was ancient. It's possible that vgo should look at the latest tag (v6.0.0 in this case), see that it has no go.mod, and then use it with a pseudo-version (v0.0.0-...) instead of picking the latest v1, but if anything explicitly asked for v1 (unlikely in this case), it would override that.

github.com/folago/nlb does not seem to be publicly visible. What did Gopkg.lock have to say about client-go? Did it specify a version or a commit or both?

@folago
Copy link
Author

folago commented Apr 3, 2018

Gopkg.lock says this on cliet-go:

[[projects]]
  name = "k8s.io/client-go"
  packages = [
    "discovery",
    "kubernetes",
    "kubernetes/scheme",
    "kubernetes/typed/admissionregistration/v1alpha1",
    "kubernetes/typed/admissionregistration/v1beta1",
    "kubernetes/typed/apps/v1",
    "kubernetes/typed/apps/v1beta1",
    "kubernetes/typed/apps/v1beta2",
    "kubernetes/typed/authentication/v1",
    "kubernetes/typed/authentication/v1beta1",
    "kubernetes/typed/authorization/v1",
    "kubernetes/typed/authorization/v1beta1",
    "kubernetes/typed/autoscaling/v1",
    "kubernetes/typed/autoscaling/v2beta1",
    "kubernetes/typed/batch/v1",
    "kubernetes/typed/batch/v1beta1",
    "kubernetes/typed/batch/v2alpha1",
    "kubernetes/typed/certificates/v1beta1",
    "kubernetes/typed/core/v1",
    "kubernetes/typed/events/v1beta1",
    "kubernetes/typed/extensions/v1beta1",
    "kubernetes/typed/networking/v1",
    "kubernetes/typed/policy/v1beta1",
    "kubernetes/typed/rbac/v1",
    "kubernetes/typed/rbac/v1alpha1",
    "kubernetes/typed/rbac/v1beta1",
    "kubernetes/typed/scheduling/v1alpha1",
    "kubernetes/typed/settings/v1alpha1",
    "kubernetes/typed/storage/v1",
    "kubernetes/typed/storage/v1alpha1",
    "kubernetes/typed/storage/v1beta1",
    "pkg/version",
    "rest",
    "rest/watch",
    "tools/cache",
    "tools/clientcmd/api",
    "tools/metrics",
    "tools/pager",
    "tools/reference",
    "transport",
    "util/buffer",
    "util/cert",
    "util/flowcontrol",
    "util/integer",
    "util/workqueue"
  ]
  revision = "78700dec6369ba22221b72770783300f143df150"
  version = "v6.0.0"

So it looks like both.

Gopkg.toml just the version:

[[constraint]]
  name = "k8s.io/client-go"
  version = "6.0.0"

@paralin
Copy link
Contributor

paralin commented Jun 7, 2018

This is also impacting us quite a bit, actually ended up switching off of client-go to solve it.

@rsc
Copy link
Contributor

rsc commented Jul 6, 2018 •
edited
Loading

Duplicate of #26328.
Duplicate of #26238.

@rsc rsc closed this as completed Jul 6, 2018
@danderson
Copy link
Contributor

The referenced issue doesn't exist. Did you mean to dupe something else, or reference a different github project?

@rsc
Copy link
Contributor

rsc commented Jul 10, 2018

Sorry, #26238. I have some kind of recurring number-rotating problem in typing issue numbers.

@golang golang locked and limited conversation to collaborators Jul 10, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
vgo
Development

No branches or pull requests
7 participants
@danderson @rsc @paralin @kardianos @folago @gopherbot @yiyus