Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: mod tidy reports toolchain not available with 'go 1.21' #62278

Open
matthewhughes-uw opened this issue Aug 25, 2023 · 35 comments
Open

cmd/go: mod tidy reports toolchain not available with 'go 1.21' #62278

matthewhughes-uw opened this issue Aug 25, 2023 · 35 comments
Assignees
Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. GoCommand cmd/go
Milestone

Comments

@matthewhughes-uw
Copy link

matthewhughes-uw commented Aug 25, 2023

Note: I think this is a question of: is "go 1.21" a valid go directive? since there was no 1.21 release, only 1.21.0, perhaps not

What version of Go are you using (go version)?

$ go version
go version go1.21.0 linux/amd64

Does this issue reproduce with the latest release?

Yes, reproduced on Go 1.21.0

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/root/.cache/go-build'
GOENV='/root/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.0'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='0'
GOMOD='/proj/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -fno-caret-diagnostics -Qunused-arguments -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build435086183=/tmp/go-build -gno-record-gcc-switches'

What did you do?

Given a basic go.mod

$ cat go.mod 
module example.com/foo

go 1.21

Then run:

$ GOTOOLCHAIN="go1.20+auto" go mod tidy
go: downloading go1.21 (linux/amd64)
go: download go1.21 for linux/amd64: toolchain not available

What did you expect to see?

go mod tidy runs successfully. I guess I expect it would pick go1.21.0 as the toolchain if no toolchain is available as 1.21

What did you see instead?

The error output above with a non-zero exit code

More details

The issue was seen when running dependabot on a repo with a go 1.21 directive in go.mod, the GOTOOLCHAIN above was taken from their usage https://github.com/dependabot/dependabot-core/blob/08ac25ebd773cede0c00be9a98e5bb03b680870b/go_modules/Dockerfile#L34

Running the above command with GODEBUG=http2debug=1 I see it:

  • requests go.dev/dl/mod/golang.org/toolchain/@v/v0.0.1-go1.21.linux-amd64.zip which returns Location: https://go.dev/dl/mod/golang.org/toolchain/@v/v0.0.1-go1.21.linux-amd64.zip
  • requesting that URL then returns location: https://dl.google.com/go/v0.0.1-go1.21.linux-amd64.zip
  • requesting that URL gives a 404

Updating the directive to: go 1.21.0 will permit things to run fine.

@matthewhughes-uw matthewhughes-uw changed the title go mod tidy reports toolchain not available with 'go 1.21' go: mod tidy reports toolchain not available with 'go 1.21' Aug 25, 2023
@matthewhughes-uw matthewhughes-uw changed the title go: mod tidy reports toolchain not available with 'go 1.21' cmd/go: mod tidy reports toolchain not available with 'go 1.21' Aug 25, 2023
@seankhliao
Copy link
Member

how did you end up with go 1.21 as a directive?

@matthewhughes-uw
Copy link
Author

matthewhughes-uw commented Aug 25, 2023

how did you end up with go 1.21 as a directive?

I can't remember exactly, but I think it was rather manual like: go1.21.0 mod edit -go=1.21

@laboger
Copy link
Contributor

laboger commented Aug 25, 2023

I am seeing this problem on linux/ppc64le.
I have built a toolchain from the latest go1.21 branch and set up my PATH to use it.
When I cd'ed to cmd/compile/internal/ssa in a toolchain from master that changes the behavior to what is shown above.
I found it when I tried to do 'go generate' but I get the same behavior from just doing 'go version' from within that directory.
If I cd back to another directory then it provides the correct go version.

$ go version
go version go1.21.0 linux/ppc64le
$ cd ~/golang/plain/go/src/cmd/compile/internal/ssa
$ pwd
/home/boger/golang/plain/go/src/cmd/compile/internal/ssa
$ go version
go: downloading go1.22 (linux/ppc64le)
go: download go1.22 for linux/ppc64le: toolchain not available
$ cd ~/gotests
$ go version
go version go1.21.0 linux/ppc64le

@bcmills
Copy link
Contributor

bcmills commented Aug 25, 2023

@matthewhughes-uw, this is as expected: go 1.21 was a development version of the language, for which there is no downloadable release (because it is not a specific version). The release version would be go 1.21.0.

In general if you want to specify a development version in the go line, you must also give a concrete toolchain version. So either of these should work:

go 1.21.0

or

go 1.21
toolchain go1.21.0

@bcmills
Copy link
Contributor

bcmills commented Aug 25, 2023

@laboger, the problem you are seeing is similar. go1.21.0 cannot upgrade automatically to a toolchain that supports go 1.22, because no such toolchain has been released. In order to compile a package in a go 1.22 module, you need to use a development build of the toolchain that supports that language version.

In particular, if you are working within $GOROOT/src you should be using exactly the toolchain built by make.bash within that GOROOT — you may need to re-run make.bash and/or check your $PATH. 😅

@bcmills bcmills added this to the Unplanned milestone Aug 25, 2023
@bcmills bcmills added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Aug 25, 2023
@bcmills
Copy link
Contributor

bcmills commented Aug 25, 2023

As far as I can tell this is all working as designed. @matthewhughes-uw, are there specific documentation changes that would have helped you solve or avoid this problem?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 25, 2023
@matthewhughes-uw
Copy link
Author

matthewhughes-uw commented Aug 25, 2023

As far as I can tell this is all working as designed

👍

@matthewhughes-uw, are there specific documentation changes that would have helped you solve or avoid this problem?

I think the main issue was my assumption that, previously I could just specify a language version (per https://go.dev/doc/toolchain#version) i.e. go 1.X in my go.mod e.g. go 1.20 but with Go 1.21 I can no longer do this. The docs say:

The version must be a valid Go release version, such as 1.9, 1.14, or 1.21rc1

and within the linked page

The syntax ‘1.N’ is called a “language version”. It denotes the overall family of Go releases implementing that version of the Go language and standard library.

so I guess it wasn't clear to me that a "language version" != a "Go release version" (though it was true for every language version before Go 1.21)

In general if you want to specify a development version in the go line, you must also give a concrete toolchain version. So either of these should work:

🤔 Maybe it's worth documenting something like "If you want to specify a 'language version' in go.mod you should (must? But again, only true for Go 1.21 at the moment) also specify a toolchain version"?

@bcmills bcmills removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Aug 25, 2023
lizthegrey added a commit to lizthegrey/tor-fetcher that referenced this issue Aug 27, 2023
damyan added a commit to ironcore-dev/ipxe that referenced this issue Aug 28, 2023
damyan added a commit to ironcore-dev/FeDHCP that referenced this issue Aug 28, 2023
Set golang version to `v1.21.0`, s. golang/go#62278 (comment)
prymitive added a commit to cloudflare/pint that referenced this issue Aug 29, 2023
@prymitive
Copy link

I think the main issue was my assumption that, previously I could just specify a language version (per https://go.dev/doc/toolchain#version) i.e. go 1.X in my go.mod e.g. go 1.20 but with Go 1.21 I can no longer do this.

To add to this: what really happened here that causes a bit of confusion is that the value of go keyword in go.mod changed it's format.

Before 1.21 release X.Y.Z value (1.20.1) would be invalid so you couldn't put it there.

With 1.21 release you most likely want to put X.Y.Z there since most user would want to put there a real working version, not a dev release.
To add to confusion X.Y value technically works, but it may or may not depending on what was released.

So the way I see it we went from never put .0 to always put .0 (0 being point release number).

Having now also toolchain (which from docs sounds like it's optional) just means that people need to wrap their heads around more complicated set of options to set, as per example from #62278 (comment).

tklauser added a commit to cilium/cilium that referenced this issue Aug 30, 2023
According to [1] as of Go 1.21 we either need to specify the full
toolchain version in the `go` directive or add a `toolchain` directive
with the concrete toolchain version. Opt for the former and make sure
it's kept up to date by renovate bot.

[1] golang/go#62278 (comment)

Signed-off-by: Tobias Klauser <tobias@cilium.io>
tklauser added a commit to cilium/cilium that referenced this issue Aug 30, 2023
According to [1] as of Go 1.21 we either need to specify the full
toolchain version in the `go` directive or add a `toolchain` directive
with the concrete toolchain version. Opt for the former and make sure
it's kept up to date by renovate bot.

[1] golang/go#62278 (comment)

Signed-off-by: Tobias Klauser <tobias@cilium.io>
@matloob
Copy link
Contributor

matloob commented Apr 17, 2024

After further consideration and discussion with @hyangah @rsc and @samthanawalla we've decided to change the behavior so that if the go.mod lists go 1.X and the current go version is less than that version, we'll attempt to download go 1.X.0 if it exists.

Many of our users are used to manually writing go 1.X in their go.mod files so it seems reasonable to allow that to continue to have a reasonable meaning. Though once this is fixed and released in 1.23 the fix will only be available to users with local toolchains of 1.23 and later.

@matloob matloob reopened this Apr 17, 2024
Quiroptero added a commit to Quiroptero/hugo-shortcodes that referenced this issue Apr 18, 2024
psFried added a commit to estuary/data-plane-gateway that referenced this issue Apr 18, 2024
See: golang/go#62278 (comment)
I have no idea why this was working in CI.
psFried added a commit to estuary/data-plane-gateway that referenced this issue Apr 18, 2024
See: golang/go#62278 (comment)
I have no idea why this was working in CI.
mx-psi pushed a commit to open-telemetry/opentelemetry-collector-contrib that referenced this issue Apr 19, 2024
**Description:** <Describe what has changed.>
<!--Ex. Fixing a bug - Describe the bug and how this fixes the issue.
Ex. Adding a feature - Explain what this achieves.-->

We currently use `go 1.21` in all go.mod files, this PR changes all
go.mod files to include the minor version by using `go 1.21.0`. It seems
that using the minor version is recommended by the Go project:
golang/go#62278. One of the dependencies in
collector-contrib also uses `go 1.21.0`, so this will need to be updated
eventually anyways: https://github.com/cilium/ebpf/blob/main/go.mod#L3.

**Link to tracking Issue:** <Issue number if applicable>

**Testing:** <Describe what testing was performed and which tests were
added.>

**Documentation:** <Describe the documentation added.>

---------

Co-authored-by: Antoine Toulme <antoine@toulme.name>
@haruyama480
Copy link
Contributor

haruyama480 commented Apr 21, 2024

@matloob

Thanks for the discussion!

I am wondering about a corner case. If this has already been discussed, please disregard.

Can we build a module having go 1.X with the toolchain go1.Xrc1?
Note that the order of go versions is like 1.X < 1.Xrc1 < 1.X.0 when X>=21.
If it works, there are difference between go 1.X and go 1.X.0.
If it doesn't work, using release candidates requires go 1.Xrc1 or toolchain go1.Xrc1.

@Alonekkpthpttall

This comment was marked as resolved.

@gopherbot
Copy link

Change https://go.dev/cl/580217 mentions this issue: Require patch in GOTOOLCHAIN or download 1.X.0 otherwise.

@samthanawalla
Copy link
Contributor

@haruyama480
Good question!

We will use 1.X.0 only if we need to upgrade & download the toolchain.

So in the case of go 1.X and toolchain go1.Xrc1, go1.Xrc1 is newer which will be used so we will not use 1.X.0

With this change, in the case of go 1.700 and toolchain=none, we will download go1.700.0 instead of go1.700

Hopefully that addresses your question.

@samthanawalla samthanawalla self-assigned this Apr 22, 2024
@samthanawalla samthanawalla added FixPending Issues that have a fix which has not yet been reviewed or submitted. and removed NeedsFix The path to resolution is known, but the work has not been done. labels Apr 22, 2024
@haruyama480
Copy link
Contributor

@samthanawalla

I understand

  • This doesn't change go version ordering though the minimum toolchain's version to build can be different between 1.X and 1.X.0
  • If we want to use a new release candidate we can explicitly specify go 1.Xrc1 and GOTOOLCHAIN=1.Xrc1

It seems to work well.

@ldemailly
Copy link

why make 1.X mean 1.X.0 instead of the more useful 1.X.p where p is the latest patch for 1.X

@samthanawalla
Copy link
Contributor

@haruyama480
Correct, version ordering should not be affected.
Correct, release candidates should still work as expected.

@ldemailly
We discussed this approach but ultimately decided on 1.X.0 to be in the spirit of reproducible builds.

shanduur pushed a commit to linode/linode-cosi-driver that referenced this issue Apr 26, 2024
## Description
<!--
What code changes are made?
What problem does this PR addresses, or what feature this PR adds?
-->
My go can't download toolchain:

```bash
go env
go: downloading go1.22 (linux/amd64)
go: download go1.22 for linux/amd64: toolchain not available
```
Here is the answer why:
golang/go#62278 (comment)

<!--
Usage: `Resolves #<issue number>`, or `Resolves <link to the issue>`.
If PR is about `failing-tests`, please post the related tests in a
comment and do not use `Resolves`
-->
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
FixPending Issues that have a fix which has not yet been reviewed or submitted. GoCommand cmd/go
Projects
None yet
Development

No branches or pull requests