Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: vgo build error "missing go.mod" #23983

Closed
robertlestak opened this issue Feb 21, 2018 · 12 comments
Closed

x/vgo: vgo build error "missing go.mod" #23983

robertlestak opened this issue Feb 21, 2018 · 12 comments
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@robertlestak
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go1.10 darwin/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

darwin/amd64

What did you do?

echo >go.mod && vgo build in project root.

What did you expect to see?

Successful build. The project builds successfully via dep with the following Gopkg.toml file:


[[constraint]]
  name = "docker.io/go-docker"
  version = "1.0.0"

[prune]
  go-tests = true
  unused-packages = true
  
[[constraint]]
  name = "github.com/google/uuid"
  version = "0.2.0"

[[constraint]]
  name = "github.com/mailgun/mailgun-go"
  version = "1.1.0"

[[constraint]]
  name = "github.com/gorilla/mux"
  version = "1.6.1"

[[constraint]]
  name = "github.com/joho/godotenv"
  version = "~1.2.0"

What did you see instead?

vgo: resolving import "github.com/lib/pq"
vgo: finding github.com/lib/pq (latest)
vgo: adding github.com/lib/pq v0.0.0-20180201184707-88edab080323
vgo: resolving import "docker.io/go-docker/api/types"
vgo: finding docker.io/go-docker (latest)
vgo: adding docker.io/go-docker v1.0.0
vgo: resolving import "github.com/docker/go-connections/nat"
vgo: finding github.com/docker/go-connections (latest)
vgo: adding github.com/docker/go-connections v0.3.0
vgo: resolving import "github.com/google/uuid"
vgo: finding github.com/google/uuid (latest)
vgo: adding github.com/google/uuid v0.1
vgo: resolving import "github.com/mailgun/mailgun-go"
vgo: finding github.com/mailgun/mailgun-go (latest)
vgo: adding github.com/mailgun/mailgun-go v1.1.0
vgo: resolving import "github.com/gorilla/mux"
vgo: finding github.com/gorilla/mux (latest)
vgo: adding github.com/gorilla/mux v1.6.1
vgo: resolving import "github.com/joho/godotenv/autoload"
vgo: finding github.com/joho/godotenv (latest)
vgo: adding github.com/joho/godotenv v1.2.0
vgo: resolving import "github.com/rs/cors"
vgo: finding github.com/rs/cors (latest)
vgo: adding github.com/rs/cors v1.2
vgo: resolving import "github.com/docker/go-units"
vgo: finding github.com/docker/go-units (latest)
vgo: adding github.com/docker/go-units v0.3.2
vgo: resolving import "github.com/opencontainers/image-spec/specs-go/v1"
vgo: finding github.com/opencontainers/image-spec (latest)
vgo: adding github.com/opencontainers/image-spec v1.0.1
vgo: resolving import "github.com/gogo/protobuf/proto"
vgo: finding github.com/gogo/protobuf (latest)
vgo: adding github.com/gogo/protobuf v1.0.0
vgo: resolving import "github.com/docker/distribution/reference"
vgo: finding github.com/docker/distribution (latest)
vgo: adding github.com/docker/distribution v0.0.0-20180209234904-6664ec703991
vgo: resolving import "golang.org/x/net/proxy"
vgo: finding golang.org/x/net (latest)
vgo: adding golang.org/x/net v0.0.0-20180218175443-cbe0f9307d01
vgo: resolving import "github.com/pkg/errors"
vgo: finding github.com/pkg/errors (latest)
vgo: adding github.com/pkg/errors v0.8.0
vgo: resolving import "github.com/opencontainers/go-digest"
vgo: finding github.com/opencontainers/go-digest (latest)
vgo: adding github.com/opencontainers/go-digest v1.0.0-rc1
vgo: finding gopkg.in/square/go-jose.v1/json v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/json v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: missing go.mod
@gopherbot gopherbot added this to the vgo milestone Feb 21, 2018
@jaloren
Copy link

jaloren commented Feb 21, 2018

Turns out if you don't have the import comment at the top, then vgo doesn't work at least initially. Did you add that to your main package before running vgo? Note there's definitely a usability issue here regardless.

package main // import "github.com/you/hello"

import (
	"fmt"
	"rsc.io/quote"
)

func main() {
	fmt.Println(quote.Hello())
}

@robertlestak
Copy link
Author

Turns out if you don't have the import comment at the top, then vgo doesn't work at least initially. Did you add that to your main package before running vgo?

Yes, the package has the import comment at the top:

$ head -n 1 main.go outputs:

package main // import "github.com/[you]/[package-name]"

$ echo >go.mod && vgo build produces the "missing go.mod" error from above, and cat go.mod produces:

module "github.com/[you]/[package-name]"

So unless vgo is dynamically getting the author and package name by other means, I would assume it is properly pulling that from the // import "github... from the main package.

@kardianos
Copy link
Contributor

kardianos commented Feb 22, 2018

It looks like it might not be handling the gopkg.in/square/go-jose.v1/json import correctly, and complaining that go-jose.v1/json is missing go.mod.

Can you make a minimal repro case that just imports this or another import and see if you can duplicate and post as a gist?

@robertlestak
Copy link
Author

I just did some testing and importing just gopkg.in/square/go-jose.v1/json or just gopkg.in/square/go-jose.v1/cipher (the two packages throwing the missing go.mod error) works fine, and it even tags them properly, whereas when built in the project altogether, it is using the pseudo-version as seen in the output above.

I manually set the pseudo-version to the version from the error output - since the go-jose.v1/json packages are farther up the dependency graph I assumed they may not have been updated by the authors of the packages I'm directly importing - but even when building with the exact commit that throws an error in the overall project, it builds fine on its own.

I'll do some more testing over the weekend.

@robertlestak
Copy link
Author

robertlestak commented Feb 25, 2018

After some further testing, I was able to narrow this down. When importing the packages gopkg.in/square/go-jose.v1/json and gopkg.in/square/go-jose.v1/cipher directly, there is no error.

Importing a package which imports these packages works as well (whether that package has been configured with a go.mod file or not, it will build regardless), however in the docker/distribution package, the import will fail with the "missing go.mod" error.

The following code will fail, as github.com/docker/distribution imports gopkg.in/square/go-jose.v1/json at distribution/vendor/gopkg.in/square/go-jose.v1/json_fork.go:22:

package main // import "github.com/you/dockerdist"

import (
	"fmt"

	"github.com/docker/distribution/uuid"
)

func main() {
	id := uuid.Generate()
	fmt.Println(id)
}

While importing the package directly works fine:

package main // import "github.com/you/josejson"

import (
	"fmt"

	"gopkg.in/square/go-jose.v1/json"
)

func main() {
	v := make(map[string]string)
	v["hello"] = "vgo"
	v["goodbye"] = "dep"
	b, err := json.Marshal(&v)
	if err != nil {
		panic(err)
	}
	fmt.Println(string(b))
}

I believe this may be related to the fact that docker/distribution uses github.com/lk4d4/vndr, but I am currently not able to replicate this issue even when using vndr.

@sigma
Copy link
Contributor

sigma commented Mar 2, 2018

I have a fairly small example that reproduces the bug:

package main // import "github.com/sigma/plop"
import _ "rsc.io/letsencrypt"

func main() {}
module "github.com/sigma/plop"
require "rsc.io/letsencrypt" v0.0.0-20160420184355-a019c9e6fce0

With that, I obtain the following when I vgo build

vgo: finding rsc.io/letsencrypt v0.0.0-20160420184355-a019c9e6fce0
vgo: finding gopkg.in/square/go-jose.v1/json v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/json v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1 v0.0.0-20160329203311-40d457b43924
vgo: finding github.com/xenolf/lego v0.0.0-20160328162834-ca19a90028e2
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160329203311-40d457b43924: missing go.mod
vgo: missing go.mod

If I look in the cache, I see this:

//vgo 0.0.3

module "rsc.io/letsencrypt"

require (
	"github.com/xenolf/lego" v0.0.0-20160328162834-ca19a90028e2
	"gopkg.in/square/go-jose.v1" v0.0.0-20160329203311-40d457b43924
	"gopkg.in/square/go-jose.v1/cipher" v0.0.0-20160329203311-40d457b43924
	"gopkg.in/square/go-jose.v1/json" v0.0.0-20160329203311-40d457b43924
)

If I manually remove those last 2 lines, then vgo proceeds to building (and fails because the libraries are outdated, but that's fair). I haven't tried to see where those spurious lines come from.

@kardianos
Copy link
Contributor

The last lines probably come from a vendor.json file where package versions don't match within a single logical module.

@sigma
Copy link
Contributor

sigma commented Mar 2, 2018

That doesn't seem to be the case. Here's the point at which vgo starts working:
rsc/letsencrypt@76104d2...33926fa#diff-bd290170e2912d3e8694db1a151066e5

vendor.json was consistent before and after, but there are other potentially significant differences: revision goes from hash to tag (random idea: maybe mapping a gopkg.in "v1" to some v0.0.0-something is buggy?), overall vendor.json format changes slightly. Will keep digging :)

@sigma
Copy link
Contributor

sigma commented Mar 2, 2018

I think the following proves the version theory? (and provides a smaller repro)

$ git clone git@github.com:rsc/letsencrypt ; cd letsecnrypt

$ GOPATH=`pwd`/gopath vgo build
vgo: finding gopkg.in/square/go-jose.v1 v1.1.0-gopkgin-v1.1.0
vgo: finding golang.org/x/time v0.0.0-20170424234030-8be79e1e0910
vgo: finding golang.org/x/net v0.0.0-20170628083913-8663ed5da4fd
vgo: finding golang.org/x/crypto v0.0.0-20170628150144-84f24dfdf3c4
vgo: finding github.com/xenolf/lego v0.0.0-20170618175828-28ead50ff1ca
vgo: finding github.com/miekg/dns v0.0.0-20170604123008-e78414ef7560
vgo: downloading github.com/xenolf/lego v0.0.0-20170618175828-28ead50ff1ca
vgo: downloading github.com/miekg/dns v0.0.0-20170604123008-e78414ef7560
vgo: downloading golang.org/x/crypto v0.0.0-20170628150144-84f24dfdf3c4
vgo: downloading golang.org/x/net v0.0.0-20170628083913-8663ed5da4fd
vgo: downloading gopkg.in/square/go-jose.v1 v1.1.0-gopkgin-v1.1.0
vgo: downloading golang.org/x/time v0.0.0-20170424234030-8be79e1e0910

# replace version by corresponding hash
$ sed -i 's/v1.1.0/aa2e30fdd1fe9dd3394119af66451ae790d50e0d/' vendor/vendor.json

$ rm -rf gopath go.mod

$ GOPATH=`pwd`/gopath vgo build                                                 
vgo: finding gopkg.in/square/go-jose.v1/json v0.0.0-20160923000811-aa2e30fdd1fe
vgo: gopkg.in/square/go-jose.v1/json v0.0.0-20160923000811-aa2e30fdd1fe: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160923000811-aa2e30fdd1fe
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160923000811-aa2e30fdd1fe: missing go.mod
vgo: finding gopkg.in/square/go-jose.v1 v0.0.0-20160923000811-aa2e30fdd1fe
vgo: finding golang.org/x/time v0.0.0-20170424234030-8be79e1e0910
vgo: finding golang.org/x/net v0.0.0-20170628083913-8663ed5da4fd
vgo: finding golang.org/x/crypto v0.0.0-20170628150144-84f24dfdf3c4
vgo: finding github.com/xenolf/lego v0.0.0-20170618175828-28ead50ff1ca
vgo: finding github.com/miekg/dns v0.0.0-20170604123008-e78414ef7560
vgo: finding gopkg.in/square/go-jose.v1/cipher v0.0.0-20160923000811-aa2e30fdd1fe
vgo: gopkg.in/square/go-jose.v1/cipher v0.0.0-20160923000811-aa2e30fdd1fe: missing go.mod
vgo: missing go.mod

@sigma
Copy link
Contributor

sigma commented Mar 2, 2018

So the problem seems to be that with proper version numbers, vgo ends up Stat()-ing URLs like: https://api.github.com/repos/square/go-jose/git/refs/tags/cipher/v1.1.0 which return an error, and makes it skip sub-packages.
When instead we have a hash, the same URL is pinged for the main package and all its subpackages, which convinces vgo that those are valid.

I workaround this locally with the following patch. That gets me going, but I strongly suspect it's not correct

diff --git a/vendor/cmd/go/internal/modfetch/gopkgin.go b/vendor/cmd/go/internal/modfetch/gopkgin.go
index f3a5987..0cc105a 100644
--- a/vendor/cmd/go/internal/modfetch/gopkgin.go
+++ b/vendor/cmd/go/internal/modfetch/gopkgin.go
@@ -92,6 +92,9 @@ func (r *gopkgin) Tags(prefix string) ([]string, error) {
 }
 
 func (r *gopkgin) Stat(rev string) (*codehost.RevInfo, error) {
+       if r.subdir != "" {
+               return nil, fmt.Errorf("subdirs won't work with gopkg.in")
+       }
        ghRev, err := r.unconvert(rev)
        if err != nil {
                return nil, err

@rsc
Copy link
Contributor

rsc commented Apr 2, 2018

The problem is that gopkg.in/square/go-jose.v1/cipher is not a module at all; gopkg.in/square/go-jose.v1 is. https://github.com/rsc/letsencrypt/blob/master/vendor/vendor.json says

		{
			"path": "gopkg.in/square/go-jose.v1/cipher",
			"revision": "v1.1.0",
			"revisionTime": "2016-09-22T17:08:11-07:00"
		},
		{
			"path": "gopkg.in/square/go-jose.v1/json",
			"revision": "v1.1.0",
			"revisionTime": "2016-09-22T17:08:11-07:00"
		}

and it sounds like those are being taken as module paths instead of as plain import paths.

@rsc rsc added the NeedsFix The path to resolution is known, but the work has not been done. label Apr 2, 2018
@gopherbot
Copy link

Change https://golang.org/cl/120042 mentions this issue: cmd/go/internal/modfetch: fix Lookup, Import; add ImportRepoRev

@golang golang locked and limited conversation to collaborators Jun 27, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

6 participants