Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: provide a way to resolve an import path to a vcs, url, etc. #18387

Open
josharian opened this issue Dec 20, 2016 · 12 comments
Open

cmd/go: provide a way to resolve an import path to a vcs, url, etc. #18387

josharian opened this issue Dec 20, 2016 · 12 comments
Labels
FeatureRequest GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@josharian
Copy link
Contributor

josharian commented Dec 20, 2016

'go get' contains non-trivial logic to figure out how to obtain a package based on its import path, including consulting a list of well-known hosting providers and checking for html meta tags. There is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go.

I'd like a way to get this information from cmd/go. Ideally it would be a library, a la package go/build, but a command line invocation would work as well, perhaps something accepting a format flag:

$ go get -resolve '{{.ImportPath}} {{.URL}} {{.IsCustom}} {{.VCS}} {{.Root}} and probably some vcs commands...' custom.xyz/pkg

or perhaps emitting JSON, with one entry per import path.

I asked for this originally at #18119 (comment). I noted there that @rogpeppe also wants this; see https://go-review.googlesource.com/8725. Another partial copy of this bit of cmd/go can be found at https://github.com/sdboyer/gps (e.g. https://github.com/sdboyer/gps/blob/master/discovery.go).

@josharian josharian added the GoCommand cmd/go label Dec 20, 2016
@josharian josharian added this to the Proposal milestone Dec 20, 2016
@rakyll
Copy link
Contributor

rakyll commented Dec 20, 2016

I'd love to see a go/get package a la go/build. It is a major blocker for custom tools that want to support go-get and not willing to shell out.

@dmitshur
Copy link
Contributor

dmitshur commented Feb 2, 2017

Edit in 2023: Please be mindful that this comment was written in 2017, when GOPATH mode was only mode that existed, before module support was added in late 2018. Its advice doesn't apply as is today.

a way to resolve an import path to a vcs, url, etc.

There is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go. ... Ideally it would be a library

Please correct me if I'm misunderstanding what this issue is about (I'm surprised no one has said anything by now).

This very functionality already exists, and has existed for a long time, in golang.org/x/tools/go/vcs package, under the func named RepoRootForImportPath.

For example, here's output from running vcs.RepoRootForImportPath("rsc.io/pdf/pdfpasswd", false):

&vcs.RepoRoot{
	VCS:  (*vcs.Cmd)(vcs.vcsGit),
	Repo: (string)("https://github.com/rsc/pdf"),
	Root: (string)("rsc.io/pdf"),
}

Note that it correctly resolves a vanity import path to "https://github.com/rsc/pdf", and the repo root to "rsc.io/pdf". It's the same code that cmd/go uses for go get, just copied to another library where it can be imported.

I know this as someone who has sent multiple CLs in order to keep the two in sync (people sometimes send a fix to one, but forget the other).

There's also issue #11490 (/cc @adg) that tracks a possible unification of the two identical sets of code. It wasn't possible before because it was hard to have something both in cmd/go and another library. But I believe that may be changing as of #18653.

@sdboyer
Copy link
Member

sdboyer commented Feb 2, 2017

Just seeing this - thanks @shurcooL for the pointer to it.

I'd also like very much to see a package like this. In fact, that's kinda what i wrote gps to be. Though, obviously, it can't be that in any official way unless/until dep makes it into the toolchain.

@bcmills
Copy link
Contributor

bcmills commented Jan 18, 2019

This very functionality already exists, and has existed for a long time, in golang.org/x/tools/go/vcs package, under the func named RepoRootForImportPath.

I would rather consolidate this functionality into go list than maintain a fork in x/tools.

We already resolve new modules in some go list commands (for example, go list -m -versions), so it seems relatively harmless to add origin information there too.

@mvdan
Copy link
Member

mvdan commented Jun 16, 2021

As per #18387 (comment), do we all agree that it's okay for go list -m to gain a way to show the VCS type and URL? Intuitively, I imagine that could be some new fields in go list -m -json, but I don't know if it would need to be opted into via some flag or require extra work.

I would love for us to converge around a specific design, so we can look at implementing it for 1.18 once the tree reopens.

Also, I'm moving this thread out of the proposal milestone, because it is not a proposal as far as I can tell.

@bcmills
Copy link
Contributor

bcmills commented Jun 24, 2022

This probably requires #44742. If and when that is addressed, we can separately consider adding the metadata to go list.

@mvdan
Copy link
Member

mvdan commented Dec 8, 2022

Interesting: I can get the @v/$version.info file from a module, which includes VCS information, but there's no easy way for me to tell what the current VCS information is. The closest I can use is @v/latest.info, but that may lag behind if the latest semver tag happened weeks or months ago.

Similar to how we have @v/list, perhaps there could be @v/info. Otherwise the new command described in #18387 (comment) give out of date information.

fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the users environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setup at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issue by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json`.

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setup at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issue by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json`.

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
@hyangah
Copy link
Contributor

hyangah commented Mar 29, 2023

Can someone summarize what is the remaining work to close this?

With go list:

@dmitshur wrote a detailed instruction in #57051

$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }

I wish I knew an easy way to resolve the mapping from a package path to its module path (ideally without creating a work module). However, that's doable.

With module proxy api:

If you know the module path and the module proxy fetched the latest version after go 1.19, you can access the similar info from /@latest endpoint.

https://proxy.golang.org/<the_module_path>/@latest

@bcmills
Copy link
Contributor

bcmills commented Mar 31, 2023

I believe that recipe only works for module paths, whereas this issue is requesting a mechanism for package import paths.

I think it is still true that “[t]here is no way to get this information other than actually doing the 'go get', or maintaining a non-trivial fork of a subset of cmd/go”, although what's left is probably a smaller change in go list (to add support for package queries) or similar.

jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in
#6938 that there
_is_ both an upstream copy
(golang/go#18387 (comment)),
which may potentially get deprecated
(golang/go#57051). Furthermore, there _is_
actually a `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051 and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
@jeffwidman
Copy link
Contributor

While this issue is clearly asking for "import path" which maps to go-import tag, I suspect some folks subscribed to this are actually looking for the "source code URL" which may be mapped to a different location via the go-source tag:

@andydotxyz
Copy link
Contributor

While this issue is clearly asking for "import path" which maps to go-import tag, I suspect some folks subscribed to this are actually looking for the "source code URL" which may be mapped to a different location via the go-source tag:

That's the case for what I am trying to do. The deprecated library suggests a go list which doesn't actually sort the use-case for what is the source root for a named package.

@andydotxyz
Copy link
Contributor

just to add a little output to summarise the challenge I am facing - given an import (/main) path the proposed solution won't work:

><> go list -m -f '{{.Origin}}' fyne.io/fyne/v2/cmd/fyne@latest
go: module fyne.io/fyne/v2/cmd/fyne: no matching versions for query "latest"

It only maps at the module root

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FeatureRequest GoCommand cmd/go NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

9 participants