Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: include vcs revision in module .info files #44742

Closed
seankhliao opened this issue Mar 2, 2021 · 7 comments
Closed

cmd/go: include vcs revision in module .info files #44742

seankhliao opened this issue Mar 2, 2021 · 7 comments

Comments

@seankhliao
Copy link
Member

It would be nice to have a way to tell which revision the proxy saw when it got a version from upstream. While this information isn't needed most of the time, it is helpful in debugging sum mismatches / moved tags. Currently to do so we can only download from both upstream and the proxy and diff the directories.

@gopherbot gopherbot added this to the Proposal milestone Mar 2, 2021
@heschi heschi changed the title proposal: proxy.golang.org: include vcs revision in .info proposal: cmd/go: include vcs revision in module .info files Mar 2, 2021
@heschi
Copy link
Contributor

heschi commented Mar 2, 2021

proxy.golang.org just forwards on the .info files generated by the go command.

cc @jayconrod @bcmills

@jayconrod
Copy link
Contributor

It wouldn't be too hard to add optional fields to .info files.

But I should point out that these files aren't authenticated, and it's possible they can change over time or be reported differently by different proxies. I think the go command would have to ignore this information when reporting security errors.

cc @matloob as well.

@seankhliao
Copy link
Member Author

I agree they shouldn't be considered for reporting security errors just for humans identifying issues, I don't think the information would be reliably available either (old versions, mod as vcs)

@hyangah
Copy link
Contributor

hyangah commented May 25, 2021

It would be easier to investigate issues like #46348 if the vcs revision info was available.

@kevinburkesegment
Copy link

I agree, this would make life a lot easier than e.g the steps that I had to take here:

argoproj/argo-rollouts#2065 (comment)

Which took about 1-2 hours of troubleshooting to pin down. Maybe someone who is more familiar would be faster but that was not me. In particular the Go command abstracts a lot and cleans up downloaded files, so it's not easy to verify, what is the proxy URL being hit, what is the command being run to fetch data from the upstream, etc.

@mvdan
Copy link
Member

mvdan commented Jul 6, 2022

Did #53644 solve this? In particular, see: https://go-review.googlesource.com/c/go/+/411397

@rsc
Copy link
Contributor

rsc commented Aug 10, 2022

This is done. If you run with GOPROXY=off you will see them in all your info files that come from git.
If you are using a proxy then it depends on what the proxy serves.
By default Go uses proxy.golang.org, which serves the vcs info in info files it has gathered since Go 1.19 was released.
For example: https://proxy.golang.org/golang.org/x/build/@v/v0.0.0-20220810151148-671cb44b90c4.info
Older info files known to the proxy have not been refreshed, so you will see a mix of with and without vcs that way.

@rsc rsc closed this as completed Aug 10, 2022
@rsc rsc changed the title proposal: cmd/go: include vcs revision in module .info files cmd/go: include vcs revision in module .info files Aug 10, 2022
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the users environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setup at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issue by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json`.

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
The used '.info' files are available only as of Go version 1.19, see
golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
The used '.info' files are available only as of Go version 1.19, see
golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
The used '.info' files are available only as of Go version 1.19, see
also [1].

[1]: golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Prepare for reading the VCS info from '.info' files under
'$GOPATH/pkg/mod' which is available only as of Go version 1.19, see
also [1].

[1]: golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setup at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issue by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json`.

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 21, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Prepare for reading the VCS info from '.info' files under
'$GOPATH/pkg/mod' which is available only as of Go version 1.19, see
also [1].

[1]: golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
This ensures that the `.info` files for all modules, which
get created by the Go command line tools under '$GOPATH/pkg/mod',
contain the VCS info. When using a Go proxy this is not guaranteed and
depends on the actual proxy instance, see [1]. This comes at the price
of increased execution time.

As the user's environment may likely be configured to use a GOPROXY,
which is the default, use an empty directory to not mix-up newly cached
files with already existing ones which might have been created using a
Go proxy and therefore may lack the VCS info.

This prepares for using the Go tooling to determine the VCS info instead
of using ORT's partial reimplementation of that.

[1]: golang/go#44742 (comment)
[2]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Prepare for reading the VCS info from '.info' files under
'$GOPATH/pkg/mod' which is available only as of Go version 1.19, see
also [1].

[1]: golang/go#44742 (comment).

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
fviernau added a commit to oss-review-toolkit/ort that referenced this issue Feb 22, 2023
Previously, ORT's GoMod integration had a rudimentary
(re-)implementation of the Go toolings' logic for resolving the VCS infos
for the dependencies, covering only the more common use cases. For
example, the implementation is VCS host specific and cannot handle mono
repository setups at all, without an ugly workaround in the downloader,
see [1]. Furthermore, I expect many not yet known issues in the more
uncommon use cases.

Avoid all these issues by just relying on the VCS info resolution of the
Go tooling.

Note that as of Go 1.19, the `.info` files under '$GOPATH/pkg/mod' are
guaranteed to contain the VCS info of the modules in case no Go proxy
is used [2]. For now that information is only accessible by parsing the
files directly, but there are plans to make this information available
via the command line tools like `go list -json` [3].

Fixes: #5555.

[1]: https://github.com/oss-review-toolkit/ort/blob/1dc5c54de3630f0f1249a7ec56ce0a3ba87ac5f1/downloader/src/main/kotlin/VersionControlSystem.kt#L361-L366
[2]: golang/go#44742 (comment)
[3]: golang/go#18387

Signed-off-by: Frank Viernau <frank_viernau@epam.com>
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in
#6938 that there
_is_ both an upstream copy
(golang/go#18387 (comment)),
which may potentially get deprecated
(golang/go#57051). Furthermore, there _is_
actually a `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051 and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether. I debated leaving the framework in place in case we
later wish to extend it for other features, but decided for now to rip
it out as it's easy enough to add later. And in general we're trying to
move toward leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
jeffwidman added a commit to dependabot/dependabot-core that referenced this issue Apr 10, 2023
Over time, we've managed to thin out our custom `go` helper in favor of
native package manager tooling via `go list`, `go get`, and `go mod`.

The final remaining usage was for translating package import URLs into
the actual repository URL for fetching metadata.

The original impetus for the `github.com/Masterminds/vcs` library was
that it was a copy/paste of some code in core `go`:

 [Further context](#4448 (comment)):

> I think the code is trying to map a Go import path back to a repository so we can e.g. link to the repo in the Dependabot PR, find the CHANGELOG, link to the diff between versions, etc, etc.
> It’s not totally trivial with Go because:
> 1. they’ve got that whole vanity URL thing going on, where you have to pass `go-get=1` in to make it act like a fancy redirect to a repo
> 2. a Go import path doesn’t necessarily actually match a repo URL. For instance, I might be able to import https://github.com/go-kit/kit/tracing, but that’s not a valid URL on GitHub. The repo is [github.com/go-kit/kit](http://github.com/go-kit/kit), and the package is browsable at https://github.com/go-kit/kit/tree/master/tracing. So Go bakes in some [logic](https://github.com/golang/go/blob/95c125a44ad1a0c3e441c3214160cd7b4483e79c/src/cmd/go/internal/vcs/vcs.go#L1437) to handle that for a series of SCMs, and the Masterminds/vcs package does a [similar thing](https://github.com/Masterminds/vcs/blob/master/vcs_remote_lookup.go) too.
>
> Ideally, we’d just use whatever golang/go does, but IIRC that wasn’t importable so the next best thing was the Masterminds/vcs implementation. The Ruby code you linked to predates our usage of that Go library, and I’d guess is just a really incomplete Ruby implementation of the same thing.
> Perhaps these days a) there’s a good Ruby implementation, or b) we think the VCS list is stable enough we’d be happy to translate it to Ruby, or c) we could somehow codegen the Ruby from the canonical Go implementation? Or we just stick with that one-off Go helper?

However, I discovered after doing some digging in #6938 that there
_is both_ an upstream copy (golang/go#18387 (comment), which may potentially get deprecated
golang/go#57051) and an actual `go list` command that provides enough of what we need:
* golang/go#44742

```
$ go list -m -f '{{.Origin}}' golang.org/x/tools@latest
{git https://go.googlesource.com/tools    refs/tags/v0.3.0 502c634771c4ba335286d55fc24eeded1704f592 }
```

While this doesn't fully resolve the upstream issue:
* golang/go#18387

For our purposes it provides enough of what we need since we already
have the module path extracted from `go.mod` (golang/go#18387 (comment)).

Performing this switch allows us to completely delete the native go
helper altogether.

I debated leaving the framework in place in case we later wish to extend
it for other features, but decided for now to rip it out as it's easy
enough to add back later. And in general we're trying to move toward
leveraging native package manager tooling whenever possible.
So if we later needed an additional feature, we're probably start with a
discussion with the `go` team to see if they'd build it into the `go`
cmd tooling.
@golang golang locked and limited conversation to collaborators Aug 10, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests

8 participants