Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: add a documentation page for reproducible builds #53976

Open
mvdan opened this issue Jul 20, 2022 · 10 comments
Open

doc: add a documentation page for reproducible builds #53976

mvdan opened this issue Jul 20, 2022 · 10 comments
Labels
Documentation GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Jul 20, 2022

See #51812 (comment) and #51812 (comment). There are a number of Go users and developers who think Go supports reproducible builds out of the box, and some others who think Go doesn't fully support reproducible builds. Neither of those is true - they are supported, but only if you do it right.

We should have a documentation page (ideally on the website, like #42119) to document this. Essentially, we want the docs to cover everything that one needs to consider to make builds reproducible, such as:

  • use the -trimpath flag
  • ensure that the cgo build is reproducible (toolchain, dependencies, etc), or disable cgo
  • ensure that the version of Go used is recorded
  • ensure that all build parameters (-tags, GOOS, GOAMD64, etc) are recorded

The last two are already covered thanks to build information stamping, I think. See how go version -m on a binary built with Go 1.18 or later records practically all Go build parameters.

This umbrella issue aims to track this work, and ensure the documentation page will cover everything that's needed before we write it. Am I missing anything in the list above?

We could also simply go with a GitHub wiki page, but just like #42119, I am reluctant to go that route as wiki pages can be edited by any user, and this would be fairly important canonical advice where any patches should be carefully reviewed on Gerrit.

@beoran
Copy link

beoran commented Jul 20, 2022

How about adding a help page for the go command for this? Like go help testflags we could have go help reproducible?

@mvdan
Copy link
Member Author

mvdan commented Jul 21, 2022

That's also an option, but I think we've been gradually moving the documentation to the website. Note how go help modules now points to pages like https://go.dev/ref/mod, for example.

@beoran
Copy link

beoran commented Jul 21, 2022

Which, as a consummate terminal user, I think is not a good evolution. For an example of what i like better, git help even opens man pages. For longer topics go help could open or refer to go doc pages.

@mvdan
Copy link
Member Author

mvdan commented Jul 21, 2022

If you want to suggest a different overall strategy to documentation, I would suggest to open a new issue, because otherwise we're going to get off-topic. This thread is about adding one new documentation page.

@toothrot toothrot added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jul 22, 2022
@toothrot toothrot added this to the Backlog milestone Jul 22, 2022
@rsc
Copy link
Contributor

rsc commented Aug 17, 2022

I think probably the wiki is the right answer for the moment. We'd like to move that to Gerrit (with self-review allowed by known contributors) at some point, which will help with the "anyone can edit" part and also with search engines indexing the wiki.

@dmitshur dmitshur added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. labels Aug 17, 2022
@mvdan
Copy link
Member Author

mvdan commented Sep 8, 2023

Re-reading #51812, and looking at go help build, I seem to understand that setting -trimpath doens't affect -buildvcs. However, in that thread, @rsc says:

We're not going to make a -reproducible flag disable cgo. That's mixing two unrelated things. At that point I don't see the difference between -reproducible and -trimpath.
[...]
So probably -trimpath is the reproducibility flag.

With cgo disabled, -trimpath isn't really a reproducibility flag today. In a local build, if I do CGO_ENABLED=0 go build -trimpath, the result is not guaranteed to be reproducible - the -buildvcs information will be stamped if the local code is a git checkout and the git tool is installed, but if I downloaded the source from a zip archive (like many packagers do, for efficiency) or don't have git installed, the information will be omitted.

So it seems to me that, if we do want -trimpath to be the reproducibility flag modulo cgo, it should enforce either -buildvcs=true or -buildvcs=false for the sake of consistent and reproducible behavior. Alternatively, combining the two flags would be "the reproducibility mode", but that doesn't feel as nice to me.

Personally I would lean towards -buildvcs=false, assuming that with #50603 implemented, we would still stamp the binary with a main module version. Project authors definitely want release binaries to include their own versions, and it would be ideal for the end user to be able to reproduce those binaries easily.

@mvdan
Copy link
Member Author

mvdan commented Sep 8, 2023

Another reason to lean towards -buildvcs=false for the "canonical" way to reproducible builds is that a build via GOPROXY, e.g. go install -trimpath foo.com/bar@v1.2.3, would never be able to embed VCS information into the binary. So it would be a shame if a git clone+checkout of the same git tag, and swapping go install [...]@latest with go build, would then break reproducibility unless you also add -buildvcs=false yourself.

@quite
Copy link

quite commented Sep 8, 2023

This is a great effort. Appreciate the care about raising awareness about -buildvcs, which I mentioned back in #57120 (comment)

@quite
Copy link

quite commented Sep 8, 2023

So it would be a shame if a git clone+checkout of the same git tag, and swapping go install [...]@latest with go build, would then break reproducibility unless you also add -buildvcs=false yourself.

Much agreed. But adding various behaviour to -trimpath also feels like overloading it. At the same time I understood that one reason behind not adding -reproducible was that it ought not to modificed CGO_ENABLED, and would thus not be "complete" (for at least that reason). Perhaps documentation of the needed flags is the right way? For visibility, I would place such docs in go help build (at least).

@mvdan
Copy link
Member Author

mvdan commented Sep 8, 2023

Documenting -buildvcs=false as one extra step for a standard reproducible build would be fine with me as well, although it would be more typing. I only raise this because earlier -trimpath had been described as the reproducibility flag modulo cgo, so I'm not sure if we're parting from that. Clear docs would be necessary either way for sure.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation GoCommand cmd/go NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

6 participants