Go 1.13 tries all module prefixes in parallel, and if another error other than "not-found" happens, the query fails. Of course the root cause is on gitlab side, but I think it's reasonable to ignore all errors if at least one attempt succeeds.
A quick workaround is clearing err if found is not empty in queryPrefixModules() in cmd/go/internal/modload/query.go.

@bcmills @jayconrod

EDIT: As a reminder, please note our https://github.com/golang/go/wiki/noplusone policy. Comments like "Me too" do not add anything new to the discussion.

I think it's reasonable to ignore all errors if at least one attempt succeeds.

I could see doing that if the provided Git URL itself returned a 404 or 410 status code, but note that gitlab.com today does not:

~$ curl -sI https://gitlab.com/nonexistentuser/xyz.git | grep HTTP
HTTP/1.1 401 Unauthorized

I'm running into issues with GitLab subgroups as well, but I'm getting a different error:

$ go get -u example.com/group/subgroup/my-go-project.git
get "example.com/group/subgroup": found meta tag get.metaImport{Prefix:"example.com/group/subgroup", VCS:"git", RepoRoot:"https://example.com/group/subgroup.git"} at //example.com/group/subgroup?go-get=1
go: finding example.com/group/subgroup/my-go-project.git latest
go get example.com/group/subgroup/my-go-project.git: git ls-remote -q https://example.com/group/subgroup.git in C:\Users\dev\go\pkg\mod\cache\vcs\5af58fdf9c9bd0fd298253dde7a0add3f4e5d6be34206b0b46f6805ee365d4ea: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://example.com/group/subgroup.git/' not found

It almost seems like it's assuming that the second part of the path must be the project. Going back to 1.12 fixes it.

Edit: Nevermind, re-ran it with -v and it's the same issue mentioned above with GitLab returning the parent.

@bcmills I mean if a try on gitlab.com/group/project succeeds, we can ignore errors encountered trying gitlab.com/group and gitlab.com/group/project/submodule, no matter what they are.

@leonshaw, maybe..? The problem with that is that it's asymmetric.

If I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

The solution to that is to treat responses other than 404/410 as hard errors. Then I'll fail to fetch example.com/foo/bar, and you'll fail to fetch example.com/foo, and we'll both (consistently!) know that we don't have enough credentials to produce a consistent result.

(More generally: if servers want to avoid leaking information to unauthenticated users, then it's more robust to deny the existence of every module rather than claim the existence of every module — especially now that #29888 is fixed.)

f I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

I don't think this is possible, at least with giltab, as they don't have "directory-level permissions" but repository-level only. Probably the same for gihub.

Even in case, if a user has credentials to every repo on the server, go get will fail to get that non-existing parent, but instead of the authorization failure git ls-remote complains about "Could not read from remote repository."

> go get -v example.com/commons/pkg/multierror
get "example.com/commons/pkg": found meta tag get.metaImport{Prefix:"example.com/commons/pkg", VCS:"git", RepoRoot:"https://example.com/commons/pkg.git"} at //example.com/commons/pkg?go-get=1
get "example.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"example.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://example.com/commons/pkg/multierror.git"} at //example.com/commons/pkg/multierror?go-get=1
go get example.com/commons/pkg/multierror: git ls-remote -q https://example.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

If I have credentials for module example.com/foo containing subdirectory ./bar/baz, and you have credentials for example.com/foo/bar containing subdirectory ./baz, and we both run go get example.com/foo/bar/baz, then we'll end up with different results for the exact same query.

Is this something more fundamental? (A module can't be uniquely located by an import path.)

at least with [gitlab], as they don't have "directory-level permissions" but repository-level only. Probably the same for [github].

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

Is this something more fundamental? (A module can't be uniquely located by an import path.)

A module has a unique path. The mapping of package paths to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

A module has a unique path. The mapping of package paths to modules is not guaranteed to be 1:1, although the go command fails with an error message if any one package within the package import graph is provided by more than one module in the module graph.

Yes, this is the problem. We need a way to avoid the collision, if possible. If it's simply treated as error, example.com/foo/bar will be unusable if example.com/foo exists.

GitLab and GitHub are not the only ways to host a module. Ideally, the semantics of the go command should generalize to self-hosted domains with arbitrary structure.

Probably, self-hosted giltab used by a majority of self-hosted git systems. However, this new behavior seems to conflict not with giltab only, but gitea as well and may conflict with any system surprised by parent discovery calls. I appreciate the generalized approach and concerns about asymmetrical behavior in some theoretical cases, but the current problem if very practical one.

I have most of a solution coded up, but there is a bad interaction with proxy fallback that I still need to work out.

Even then, my expectation is that a server that wishes to indicate “there is no repository here” (for a repository explicitly mentioned in a go-import <meta> tag) will serve a 404 or a 410 for the URL. I'm not sure whether gitlab.com does that for authenticated users, but for unauthenticated users they're currently serving 401s instead.

Change https://golang.org/cl/194560 mentions this issue: cmd/go/internal/modfetch: report the module path for errors in (*codeRepo).Versions

Change https://golang.org/cl/194561 mentions this issue: cmd/go/internal/modfetch/codehost: treat nonexistent repositories as “not found”

@gopherbot, please backport to 1.13: this is a regression, and prevents previously-working modules from being fetched.

Backport issue(s) opened: #34215 (for 1.13).

Remember to create the cherry-pick CL(s) as soon as the patch is submitted to master, according to https://golang.org/wiki/MinorReleases.

I've implemented the approach described in #34094 (comment). That should solve the problem if GitLab is correctly serving 404 or 410 responses for nonexistent paths for signed-in users.

However, I don't know whether they're doing that, so please help me confirm.

@umputun (or one of the other folks with this problem): please try a go command built from head and let me know whether your problem is resolved. (The easy way to try a go command built from head is to install golang.org/x/dl/gotip, then run gotip download and use gotip in place of the regular go command.)

Does this mean that this will get fixed for Go1.14, but will still not work for 1.13?

@xavivars, this has already been fixed at head and backported to the 1.13 branch. It will be included in Go 1.13.3.

I've just tried with Go 1.13.3 but I still failed in go get my subgroup repository.

$ go version
go version go1.13.3 linux/amd64

$ go get -v gitlab.com/foo/bar/example
get "gitlab.com/foo/bar": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar?go-get=1
get "gitlab.com/foo/bar/example": found meta tag get.metaImport{Prefix:"gitlab.com/foo/bar", VCS:"git", RepoRoot:"https://gitlab.com/foo/bar.git"} at //gitlab.com/foo/bar/example?go-get=1
get "gitlab.com/foo/bar/example": verifying non-authoritative meta tag
go get gitlab.com/foo/bar/example: git ls-remote -q https://gitlab.com/foo/bar.git in /home/koala/Workspace/gopath/pkg/mod/cache/vcs/xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx: exit status 128:
	The project you were looking for could not be found.
	fatal: Could not read from remote repository.
	
	Please make sure you have the correct access rights
	and the repository exists.

P.S. My repository is a private repository. I don't know if it is a matter or not.

@yookoala, you probably need to configure a git credential helper, or add a .netrc file containing credentials. (The go command is not intended to prompt for interactive logins.)

See https://git-scm.com/docs/gitcredentials and #29953.

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

@umputun, thanks for confirming! I'm planning to leave the issue open for adding a more realistic regression test in 1.14 (probably during the freeze).

go version 1.13.3 fixed the problem. I have tried it against public (gitlab hosted) repo as well as self-hosted giltab. In both cases worked as expected. thx @bcmills and others helped to diagnose and address the problem.

should I close the ticket? This was my first ticket here and I'm not sure what the rule.

My go version:

C:\Users\my-gitlab>go version
go version go1.13.3 windows/amd64

simple description:
with private subgroup can not be go get, see below:
my-group are public
my-private-group and it's project sample all are private
my-group and it's project sample all are public
private-demo are private

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-subgroup/sample
go: finding git2.my-gitlab.net.cn/my-group/my-subgroup/sample latest
go: downloading git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9
go: extracting git2.my-gitlab.net.cn/my-group/my-subgroup/sample v0.0.0-20191022081546-6534fce037e9

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample
go get git2.my-gitlab.net.cn/my-group/my-private-subgroup/sample: git ls-remote -q origin in D:\workspace\code\go\pkg\mod\cache\vcs\8314af4c2579c0ee8b974f039f8246d179834d34c834680bfe59ce583888c820: exit status 128:
        remote: The project you were looking for could not be found.
        fatal: repository 'https://git2.my-gitlab.net.cn/my-group/my-private-subgroup.git/' not found

C:\Users\my-gitlab>go get git2.my-gitlab.net.cn/my-group/private-demo
go: finding git2.my-gitlab.net.cn/my-group/private-demo latest
go: downloading git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8
go: extracting git2.my-gitlab.net.cn/my-group/private-demo v0.0.0-20191022084509-90eb539586c8

So, have you tested private group or subgroup on gitlab?

Thanks anyway!

have you tested private group or subgroup on gitlab?

No, I didn't but tried it now with similar setup - commons group public, pkg subgroup public and syncs repo private (internal in gitlab's permissions). I see very similar results:

go version go1.13.3 darwin/amd64

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
go get git.internal.com/commons/pkg/syncs: git ls-remote -q https://git.internal.com/commons/pkg.git in /Users/umputun/go-home/pkg/mod/cache/vcs/882b5ac6f69791979e3abd70af30e871e32fe91173f2e25079ef6a88464ec13e: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.

Not sure if important, but to access private repos from go I use insteadOf rewrite:

 [url "git@git.internal.com:"]
     insteadOf = https://git.internal.com/

@umputun, thanks for the additional detail.

This line in particular:

get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg/syncs?go-get=1

seems to indicate a bug or misconfiguration on the server side. If the repo is hosted at git.internal.com/commons/pkg/syncs, then the server should reply with that as the prefix and https://git.internal.com/commons/pkg/syncs.git as the repo root (instead of the git.internal.com/commons/pkg subgroup as shown).

Did that repository work with Go 1.12?

doesn't work with 1.12 either

go version go1.12.12 linux/amd64, from golang:1.12 container:

go get -v git.internal.com/commons/pkg/syncs

Fetching https://git.internal.com/commons/pkg/syncs?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg/syncs?go-get=1 (status code 200)
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg/syncs?go-get=1
get "git.internal.com/commons/pkg/syncs": verifying non-authoritative meta tag
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
Fetching https://git.internal.com/commons/pkg?go-get=1
Parsing meta tags from https://git.internal.com/commons/pkg?go-get=1 (status code 200)
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at https://git.internal.com/commons/pkg?go-get=1
Fetching https://git.internal.com/commons?go-get=1
Parsing meta tags from https://git.internal.com/commons?go-get=1 (status code 200)
Fetching https://git.internal.com?go-get=1
Parsing meta tags from https://git.internal.com?go-get=1 (status code 200)
go get git.internal.com/commons/pkg/syncs: git ls-remote -q origin in /go/pkg/mod/cache/vcs/82b580717104e8b7bf1de06ef2b675a3ca5bf7d044f776fac4629633677f22da: exit status 128:
	> GitLab: The project you were looking for could not be found.
	fatal: Could not read from remote repository.

	Please make sure you have the correct access rights
	and the repository exists.	

However with ~/.netrc everyting seems to work (1.13.3):

go get -v git.internal.com/commons/pkg/syncs
get "git.internal.com/commons/pkg": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg.git"} at //git.internal.com/commons/pkg?go-get=1
get "git.internal.com/commons/pkg/syncs": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/syncs", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/syncs.git"} at //git.internal.com/commons/pkg/syncs?go-get=1
go: finding git.internal.com/commons/pkg/syncs v1.1.0
go: downloading git.internal.com/commons/pkg/syncs v1.1.0
go: extracting git.internal.com/commons/pkg/syncs v1.1.0
get "git.internal.com/commons/pkg/multierror": found meta tag get.metaImport{Prefix:"git.internal.com/commons/pkg/multierror", VCS:"git", RepoRoot:"https://git.internal.com/commons/pkg/multierror.git"} at //git.internal.com/commons/pkg/multierror?go-get=1
git.internal.com/commons/pkg/multierror
git.internal.com/commons/pkg/syncs	

So, looks like .gitconfig insteadOf doesn't play well with go mods, but .netrc is fine.

.netrc is what I would recommend anyway, so it sounds like the overall state is improving, at least. (.netrc wasn't working well in 1.12.)

I didn't get to the more realistic regression test during the 1.14 cycle, and by now I don't remember which cases were remaining to be tested. 😞

I'm happy to review further improvements to GitLab integration testing, but I don't think we need to leave this issue open.

GitLab Community Edition 12.3.5 self-hosted
golang 1.13.8 & 1.14
this issue still there

@Yi-Hunter, “this issue” as reported by its original author has been verified as fixed. If you're seeing a similar symptom, please open a new issue with details and steps to reproduce it.