Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: in 1.13, the way extracting module vcs address might be incorrect #34339

Closed
Ronmi opened this issue Sep 17, 2019 · 9 comments
Closed
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@Ronmi
Copy link

Ronmi commented Sep 17, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

No, go mod tidy runs smoothly in 1.12/1.11

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ronmi/.cache/go-build"
GOENV="/home/ronmi/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY="*.jen.da"
GONOSUMDB="*.jen.da"
GOOS="linux"
GOPATH="/home/ronmi/go"
GOPRIVATE="*.jen.da"
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/home/ronmi/golang/1.13"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/home/ronmi/golang/1.13/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ronmi/jenda/richpay_drivers/firstbank/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"

What did you do?

We hosted a vcs server

  1. import "git.jen.da/richpay/richpay/common"
  2. go env -w GOPRIVATE=*.jen.da
  3. go mod tidy

What did you expect to see?

It downloads correct content and writes correct record into mod cache, as in go 1.12/1.11

What did you see instead?

It downloads correct contents into $GOPATH/pkg/mod/git.jen.da/richpay/richpay@v0.0.0-20190916085810-25a948e18f6c, but write incorrect git config in mod cache.

Here's the dump of git config in mod cache ($GOPATH/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6)

[core]
        repositoryformatversion = 0
        filemode = true
        bare = true
[remote "origin"]
        url = http://git.jen.da/richpay/.git
        fetch = +refs/heads/*:refs/remotes/origin/*

As mentioned above, the origin.url is supposed to be http://git.jen.da/richpay/richpay.git

Only repo named same as user is affected, importing something like git.jen.da/richpay/tools works smoothly.

@jayconrod
Copy link
Contributor

cc @bcmills

The domain git.jen.da can't be resolved using public DNS, so it may be difficult to reproduce this.

@Ronmi Could you post the output of the commands below:

curl https://git.jen.da/richpay/richpay/common?go-get=1
curl https://git.jen.da/richpay/richpay?go-get=1
curl https://git.jen.da/richpay?go-get=1

The server should respond with a go-import meta tag that indicates where the origin VCS server is. That should more or less literally end up as the remote URL in the git repository.

@jayconrod jayconrod added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 17, 2019
@Ronmi
Copy link
Author

Ronmi commented Sep 17, 2019

Thanks for reply.

We use gitea 1.9.1 as git server

$ curl https://git.jen.da/richpay/richpay/common?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay/richpay git http://git.jen.da/richpay/richpay.git">
		<meta name="go-source" content="git.jen.da/richpay/richpay _ http://git.jen.da/richpay/richpay/src/branch/master{/dir} http://git.jen.da/richpay/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay/richpay
	</body>
</html>
$ curl https://git.jen.da/richpay/richpay?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay/richpay git http://git.jen.da/richpay/richpay.git">
		<meta name="go-source" content="git.jen.da/richpay/richpay _ http://git.jen.da/richpay/richpay/src/branch/master{/dir} http://git.jen.da/richpay/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay/richpay
	</body>
</html>
$ curl https://git.jen.da/richpay?go-get=1
<!doctype html>
<html>
	<head>
		<meta name="go-import" content="git.jen.da/richpay git http://git.jen.da/richpay/.git">
		<meta name="go-source" content="git.jen.da/richpay _ http://git.jen.da/richpay/src/branch/master{/dir} http://git.jen.da/richpay/src/branch/master{/dir}/{file}#L{line}">
	</head>
	<body>
		go get git.jen.da/richpay
	</body>
</html>

go.mod in richpay/richpay (running go mod tidy here doesn't cause problem)

module git.jen.da/richpay/richpay

go 1.12

require (
        github.com/DATA-DOG/go-sqlmock v1.3.3 // indirect
        github.com/ericlagergren/decimal v0.0.0-20190912144844-2c3e3e1ef942 // indirect
        github.com/go-sql-driver/mysql v1.4.1
        github.com/gofrs/uuid v3.2.0+incompatible // indirect
        github.com/kat-co/vala v0.0.0-20170210184112-42e1d8b61f12
        github.com/lib/pq v1.2.0 // indirect
        github.com/pkg/errors v0.8.1
        github.com/raohwork/envexist v0.0.1
        github.com/raohwork/jsonapi v0.1.0
        github.com/sirupsen/logrus v1.4.2
        github.com/spf13/viper v1.4.0
        github.com/volatiletech/inflect v0.0.0-20170731032912-e7201282ae8d // indirect
        github.com/volatiletech/null v8.0.0+incompatible
        github.com/volatiletech/sqlboiler v3.5.0+incompatible
)

go.mod in problematic module (but safe with go 1.12)

module git.jen.da/richpay_drivers/firstbank

go 1.12

result of running go mod tidy -v

get "git.jen.da/richpay/richpay/models": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models?go-get=1
get "git.jen.da/richpay/richpay/models": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay?go-get=1
get "git.jen.da/richpay/richpay/dba": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/dba?go-get=1
get "git.jen.da/richpay/richpay/dba": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/driver": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/driver?go-get=1
get "git.jen.da/richpay/richpay/models/payment/driver": verifying non-authoritative meta tag
get "git.jen.da/richpay": found meta tag get.metaImport{Prefix:"git.jen.da/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/.git"} at //git.jen.da/richpay?go-get=1
get "git.jen.da/richpay/richpay/models/payment": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment?go-get=1
get "git.jen.da/richpay/richpay/models/payment": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/common": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/common?go-get=1
get "git.jen.da/richpay/richpay/common": verifying non-authoritative meta tag
get "git.jen.da": found meta tag get.metaImport{Prefix:"git.jen.da", VCS:"git", RepoRoot:"http://git.jen.da//.git"} at //git.jen.da/?go-get=1
get "git.jen.da/richpay/richpay/models/payment/orders": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/orders?go-get=1
get "git.jen.da/richpay/richpay/models/payment/orders": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/results": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/models/payment/results?go-get=1
get "git.jen.da/richpay/richpay/dba/paymentdb": found meta tag get.metaImport{Prefix:"git.jen.da/richpay/richpay", VCS:"git", RepoRoot:"http://git.jen.da/richpay/richpay.git"} at //git.jen.da/richpay/richpay/dba/paymentdb?go-get=1
get "git.jen.da/richpay/richpay/dba/paymentdb": verifying non-authoritative meta tag
get "git.jen.da/richpay/richpay/models/payment/results": verifying non-authoritative meta tag
go: finding git.jen.da/richpay/richpay latest
go: finding git.jen.da/richpay/richpay/models latest
go: finding git.jen.da/richpay/richpay/models/payment latest
go: finding git.jen.da/richpay/richpay/common latest
go: finding git.jen.da/richpay/richpay/dba latest
go: finding git.jen.da/richpay/richpay/models/payment/orders latest
go: finding git.jen.da/richpay/richpay/dba/paymentdb latest
go: finding git.jen.da/richpay/richpay/models/payment/driver latest
go: finding git.jen.da/richpay/richpay/models/payment/results latest
go: finding github.com/volatiletech/inflect latest
git.jen.da/richpay_drivers/firstbank imports
	git.jen.da/richpay/richpay/common: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/dba/paymentdb: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/driver: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/orders: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found
git.jen.da/richpay_drivers/firstbank/vacct imports
	git.jen.da/richpay/richpay/models/payment/results: git ls-remote -q http://git.jen.da/richpay/.git in /home/ronmi/go/pkg/mod/cache/vcs/c4ba413bf2fb1e97c0b587f6f7a0f095d2fd852b4bb0216247272f4bb68ff7a6: exit status 128:
	fatal: repository 'http://git.jen.da/richpay/.git/' not found

There's no single line imports git.jen.da/richpay

$ grep -F git.jen.da *.go **/*.go|grep -vF '// import'|awk '{print $2}'|sort|uniq
"git.jen.da/richpay_drivers/firstbank"
"git.jen.da/richpay_drivers/firstbank/lib"
"git.jen.da/richpay_drivers/firstbank/libcheck"
"git.jen.da/richpay/richpay/common"
"git.jen.da/richpay/richpay/dba/paymentdb"
"git.jen.da/richpay/richpay/models/payment/driver"
"git.jen.da/richpay/richpay/models/payment/orders"
"git.jen.da/richpay/richpay/models/payment/results"

I also tried few other ways

  1. Adds require git.jen.da/richpay/richpay latest in go.mod manually, everything works.
  2. Remove the richpay line in working go.mod and re-run go mod tidy causes same problem again.
  3. switch back to go 1.12 at anytime also works.

@bcmills
Copy link
Contributor

bcmills commented Sep 17, 2019

This is likely the same underlying issue as #34094 (see #34094 (comment) in particular).

@Ronmi, please try a gotip build and see whether your issue has been resolved. (I suspect that it has.)

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 17, 2019
@Ronmi
Copy link
Author

Ronmi commented Sep 17, 2019

Thanks for the references. I'm pretty sure these are same issues, excepts one small detail:

Our gitea server requires users login before accessing any repo (by enabling REQUIRE_SIGNIN_VIEW). After disabling it, gotip fixes the problem while go 1.13 remains failed.

I tried wget https://git.jen.da/richpay/richpay.git/ and wget https://git.jen.da/richpay/.git/, seems gitea does not issue 401 if auth is required. Instead, it uses 302 to redirect user to login page, which is common strategy in web apps.

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

@bcmills
Copy link
Contributor

bcmills commented Sep 18, 2019

Our gitea server requires users login before accessing any repo (by enabling REQUIRE_SIGNIN_VIEW). After disabling it, gotip fixes the problem while go 1.13 remains failed.

Our expectation is that users are already authenticated to the server (via credentials stored in a .netrc file, or hopefully eventually using the more general GOAUTH protocol described in #26232 (comment).

So you should not have to enable REQUIRE_SIGNIN_VIEW provided that you have appropriate credentials stored in the .netrc file.

@bcmills
Copy link
Contributor

bcmills commented Sep 18, 2019

I tried wget https://git.jen.da/richpay/richpay.git/ […]. Instead, it uses 302 to redirect user to login page, which is common strategy in web apps.

As of CL 194561, there are two opportunities for an HTTPS server to signal that a given module path does not exist:

  1. It may elect not to serve a <meta name="go-import"> tag in the initial go-get=1 response, regardless of the status code.
  2. It may serve a <meta […]> tag pointing to a repository with an HTTPS URL, and then serve a 404 or 410 response for that repository.

Following approach (1), as long as the 302'd page does not include a misleading <meta […]> tag, it shouldn't matter what status code it serves for the https://[…].git URLs.

@bcmills
Copy link
Contributor

bcmills commented Sep 18, 2019

After disabling it, gotip fixes the problem while go 1.13 remains failed.

Thanks for confirming. In that case, shall I close this as a duplicate of #34094?

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

For reference, that seems to be go-gitea/gitea#8095.

@bcmills bcmills added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Sep 18, 2019
@Ronmi
Copy link
Author

Ronmi commented Sep 20, 2019

After disabling it, gotip fixes the problem while go 1.13 remains failed.

Thanks for confirming. In that case, shall I close this as a duplicate of #34094?

I'm good with it since the root cause is bug of Gitea. Thanks for your help.

I also noticed that gitea 1.9.3 fixed response of ?go-get=1 which can solve this problem as well.

For reference, that seems to be go-gitea/gitea#8095.

Yes. That's where I found the information.

@bcmills
Copy link
Contributor

bcmills commented Sep 20, 2019

Duplicate of #34094

@bcmills bcmills marked this as a duplicate of #34094 Sep 20, 2019
@bcmills bcmills closed this as completed Sep 20, 2019
@golang golang locked and limited conversation to collaborators Sep 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

4 participants