Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

doc: mention "purego" build tag convention somewhere #23172

Open
dsnet opened this issue Dec 18, 2017 · 23 comments
Open

doc: mention "purego" build tag convention somewhere #23172

dsnet opened this issue Dec 18, 2017 · 23 comments

Comments

@dsnet
Copy link
Member

dsnet commented Dec 18, 2017

As the number of Go implementations continues to increase, the number of cases where the unsafe package is unlikely to work properly also rises. Currently, there is appengine, gopherjs, and possibly wasm where pointer arithmetic is not allowed.

Currently, protobuf and other packages special cases build tags for appengine and js and may need to add others in the near future. It does not scale very well to blacklist specific known Go implementations where unsafe does not work.

My proposal is to document safe as a community agreed upon tag meaning that unsafe should not be used. It is conceivable that this concept be extended to the compiler rejecting programs that use unsafe when the safe tag is set, but I'm currently more interested as a library owner in knowing whether to avoid unsafe in my own packages.

\cc @zombiezen @dneil @neelance @shurcooL

@gopherbot gopherbot added this to the Proposal milestone Dec 18, 2017
@dsnet
Copy link
Member Author

dsnet commented Dec 18, 2017

The code history of protobuf seems to indicate that this very same concept was discussed but not pursued further. I'd like to push this more since I see this distinction in at least 2 packages I own.

golang/protobuf#154

@dsnet dsnet closed this as completed Dec 18, 2017
@dsnet dsnet reopened this Dec 18, 2017
@mdempsky
Copy link
Member

I think standardizing a build tag to indicate whether package unsafe is available makes sense.

It is conceivable that this concept be extended to the compiler rejecting programs that use unsafe when the safe tag is set

I disagree. Currently, build tags are strictly a build-system concept. I'd argue the compiler should remain ignorant of them. cmd/compile already has a -u flag that prevents importing package unsafe, and the build system can arrange to pass -u as appropriate.

@dmitshur
Copy link
Contributor

dmitshur commented Dec 19, 2017

/cc @bradfitz who also ran into this with go4.org/reflectutil, and seemed to like "safe" at the time.

@bradfitz
Copy link
Contributor

@shurcooL, I'm still fine with "safe", as long as it's defined (i.e. "code that doesn't import "unsafe").

But does it also mean no assembly?

Those are the sorts of things that should be clarified, if this is to be blessed somehow. (our own use, wiki page, etc)

@dmitshur
Copy link
Contributor

@dsnet Can you clarify if your proposal is about documenting safe to have a very specific meaning and applied to all packages?

Or is it about documenting the fact that safe is a commonly used build tag for a given purpose, but individual projects still have get final say on the exact meaning of the safe build tag for their own needs?

@flimzy
Copy link
Contributor

flimzy commented Dec 19, 2017

Would this proposal be codified in the standard library somehow, perhaps by adding a !safe build tag to the unsafe package? Or would it live purely in documentation?

@davecb
Copy link

davecb commented Dec 19, 2017

In a previous life, we had to identify versions of libraries, and rapidly found out that that was too coarse a measure. We eventually attached a label via the linker to each entry point*, and could tell if, for example, a call to memcpy allowed overap or not. '

We also used it in migration work, to identify parts of programs that could not be supported on a different OS or hardware platform.

You arguably should consider labelling parts of the unsafe library with supported and unsupported by target OS, language or whatever, not the whole library if only one operation is unavailable.

--dave
[* a description of using per-entry-point labels for a different purposes is at https://leaflessca.wordpress.com/2017/02/12/dll-hell-and-avoiding-an-np-complete-problem/ ]

@andlabs
Copy link
Contributor

andlabs commented Jan 16, 2018

Would this unsafe build tag also affect code that uses cgo? SWIG?

How would this build tag interact with the standard library, where both are used a lot? Does no unsafe mean no reflect as well?

What is the unsafe policy on nacl? is there anything in nacl that we could use for this?

@shurcooL it sounds like the latter at minimum, the former ideally.

@dsnet
Copy link
Member Author

dsnet commented Jan 16, 2018

I propose that "safe" be soft signal that a library should have memory safety (i.e., makes no assumptions about how objects are laid out in memory, the architecture endianess, semantics regarding registers, etc).

Thus, the "safe" tag has the following properties:

  • This is just a hint for library authors who want to write code that is highly portable. There is no logic in the compiler or the build tool to enforce this.
    • Thus, the standard library doesn't have to use it. Portability is achieve by either requiring a fork of the standard library (as is the case for gopherjs) or via a series of build tags as the mainline standard library does for various architectures.
  • Use of reflect is allowed since it doesn't allow you to violate memory safety.
  • Use of cgo is allowed. We already have a build tag for that, which is cgo. In practice, safe implies that cgo is not used since it is difficult to use cgo without pointers (for which you need unsafe).
  • Use of assembly is forbidden. Any reasonable use of assembly makes assumption about how memory is laid out on the stack and/or heap.

Thus, appengine and gopherjs are example toolchains that would always set the safe tag.

@rsc
Copy link
Contributor

rsc commented Jan 22, 2018

Based on discussion with proposal-review:

  • It seems reasonable for appengine, gopherjs, and wasm, all of which declare their own "restricted build" tag, to agree on a common one.
  • It should mean no asm, no cgo, no unsafe. (It's impossible to use cgo without unsafe.)
  • Nothing in the standard distribution would care; this is really about coordination between these other non-standard environments. Where do you propose to document this?
  • A better name than "safe" would be nice. "purego"?

@neelance
Copy link
Member

On WebAssembly: The wasm backend that I'm working on uses a linear memory, so unsafe is fully supported. It also has its own asm instructions, just like other architectures. Cgo is not supported (unless someone wants to do a crazy integration with emscripten).

@cznic
Copy link
Contributor

cznic commented Jan 23, 2018

A better name than "safe" would be nice. "purego"?

👍 for purego, already using it in my projects for some years.

@dsnet
Copy link
Member Author

dsnet commented Jan 23, 2018

I support purego as well. Even if WebAssembly supports unsafe (which is great to hear!), there is always still the use case where someone wants to compile with a pure-Go version for a variety of reasons.

I don't have any great suggestions for where to document this, but perhaps the godoc for go/build?

@dmitshur
Copy link
Contributor

I wanted to point out that in colloquial usage, I've seen "pure Go" most commonly refer to packages that don't use cgo (but can use unsafe, assembly). Seeing it mean "no unsafe and no assembly as well" would require some calibration. But maybe it's fine.

The math/big package contains some precedent on this: it defines a math_big_pure_go tag, which is being used as proposed here (no assembly, no unsafe, no cgo).

@rsc
Copy link
Contributor

rsc commented Jan 29, 2018

OK, purego it is.

@rsc rsc changed the title proposal: doc: agree upon a "safe" build tag proposal: doc: mention "purego" build tag convention somewhere Jan 29, 2018
@rsc rsc changed the title proposal: doc: mention "purego" build tag convention somewhere doc: mention "purego" build tag convention somewhere Jan 29, 2018
@rsc rsc modified the milestones: Proposal, Go1.11 Jan 29, 2018
cespare added a commit to cespare/xxhash that referenced this issue Jan 29, 2018
This is the convention that the Go ecosystem is standardizing on.

See golang/go#23172.
dmitshur added a commit to gopherjs/gopherjs that referenced this issue Feb 2, 2018
The proposal in golang/go#23172 was accepted. The "purego" build tag
is intended to be a community agreed upon soft-signal to indicate the
forbidden use of unsafe, assembly, or cgo.

Since GopherJS doesn't support those, it should always use "purego"
versions of Go packages when they're available. This is done by always
setting the "purego" build constraint.

Fixes #746.
dsnet added a commit to google/go-cmp that referenced this issue Feb 2, 2018
The proposal in golang/go#23172 was accepted. The "purego" build tag
is intended to be a community agreed upon soft-signal to indicate the
forbidden use of unsafe, assembly, or cgo.

A change in the future will remove special-casing the appengine and js tags
once the related toolchains support purego (possibly after some bake-in period).
ernado added a commit to ClickHouse/ch-go that referenced this issue Jan 3, 2022
* Add arm64
* Use purego instead of nounsafe

Ref: golang/go#23172
Kudos: @cristaloleg
@kortschak
Copy link
Contributor

I'd like to clarify, based on discussion here, whether this should even be a thing. I know that people want it, but it looks like in that discussion it isn't being considered as having any great importance.

chenyt9 pushed a commit to MotorolaMobilityLLC/external-golang-protobuf that referenced this issue Jul 12, 2023
The proposal in golang/go#23172 was accepted. The "purego" build tag
is intended to be a community agreed upon soft-signal to indicate the
forbidden use of unsafe, assembly, or cgo.

A change in the future will remove special-casing the appenGine and js tags
once the related toolchains support purego (possibly after some bake-in period).
@gopherbot
Copy link

Change https://go.dev/cl/561935 mentions this issue: crypto: use and test purego tag consistently

gopherbot pushed a commit that referenced this issue Mar 4, 2024
Fixes #58636
Updates #23172

Change-Id: I578a5597f467be45a7d6fb7582b24855b2e6512b
Reviewed-on: https://go-review.googlesource.com/c/go/+/561935
Reviewed-by: Roland Shoemaker <roland@golang.org>
Reviewed-by: Michael Knyszek <mknyszek@google.com>
Auto-Submit: Filippo Valsorda <filippo@golang.org>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
@aykevl
Copy link

aykevl commented Mar 15, 2024

I'm working on TinyGo, and combining these three concepts together seems like a bad idea to me (no assembly, no cgo, and no unsafe).
TinyGo:

  • Supports unsafe, though it has a slightly different memory layout for some types. It should still be possible to write portable unsafe code in most cases (e.g. unsafe.String was a great addition for this).
  • Supports CGo, though it has some features missing compared to the main Go toolchain. These missing features can probably be implemented when needed.
  • Does not support Go assembly. I've tried it, and the only way I got it to work was one giant hack.

Furthermore, there is already a perfectly good tag for CGo support: the cgo build tag. I don't think we need a new build tag that also says something about CGo support.
I see there are some systems (like appengine) where unsafe is not allowed, I would suggest using a different build tag for that than the one that controls Go assembly support.

Right now we set the purego build tag by default to get crypto packages to work, but I don't really like it because it's not very clearly defined right now and I'd rather not limit things like unsafe. A build tag like noasm and a separate build tag like nounsafe (or whatever) would be much better in my opinion.

mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Mar 22, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/circl that referenced this issue Mar 26, 2024
The `purego` build tag has been proposed[1] as the de-facto tag for
pure Go builds. This is a superset of the `noasm` tag currently used.

This patch aligns the code base to toggle to pure Go implementation
when `purego` is defined.

Verified that all assembly is excluded by doing:

    find . -name '*.s' -exec sed -i -e '$athis-is-not-valid-assembly 12345' {} \;
    go test --tags purego ./...

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/circl that referenced this issue Mar 26, 2024
The `purego` build tag has been proposed[1] as the de-facto tag for
pure Go builds. This is a superset of the `noasm` tag currently used.

This patch aligns the code base to toggle to pure Go implementation
when `purego` is defined.

Verified that all assembly is excluded by doing:

    find . -name '*.s' -exec sed -i -e '$athis-is-not-valid-assembly 12345' {} \;
    go test --tags purego ./...

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/bazel-gazelle that referenced this issue Mar 28, 2024
`purego` is a proposed[1] de-facto build constraint to
denote building _only_ Go code.

We must ignore this constraint to make sure that files
that toggle on `purego` are added to the Bazel target
`srcs` and let the compiler decide if they should be
compiled or not.

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/bazel-gazelle that referenced this issue Mar 28, 2024
`purego` is a proposed[1] de-facto build constraint to
denote building _only_ Go code.

We must ignore this constraint to make sure that files
that toggle on `purego` are added to the Bazel target
`srcs` and let the compiler decide if they should be
compiled or not.

[1]: golang/go#23172
fmeum pushed a commit to bazelbuild/bazel-gazelle that referenced this issue Mar 28, 2024
`purego` is a proposed[1] de-facto build constraint to
denote building _only_ Go code.

We must ignore this constraint to make sure that files
that toggle on `purego` are added to the Bazel target
`srcs` and let the compiler decide if they should be
compiled or not.

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 3, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 5, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 5, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 6, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 6, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 6, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/circl that referenced this issue Apr 6, 2024
The `purego` build tag has been proposed[1] as the de-facto tag for
pure Go builds. This is a superset of the `noasm` tag currently used.

This patch aligns the code base to toggle to pure Go implementation
when `purego` is defined.

Verified that all assembly is excluded by doing:

    find . -name '*.s' -exec sed -i -e '$athis-is-not-valid-assembly 12345' {} \;
    go test --tags purego ./...

[1]: golang/go#23172
armfazh pushed a commit to cloudflare/circl that referenced this issue Apr 8, 2024
The `purego` build tag has been proposed[1] as the de-facto tag for
pure Go builds. This is a superset of the `noasm` tag currently used.

This patch aligns the code base to toggle to pure Go implementation
when `purego` is defined.

Verified that all assembly is excluded by doing:

    find . -name '*.s' -exec sed -i -e '$athis-is-not-valid-assembly 12345' {} \;
    go test --tags purego ./...

[1]: golang/go#23172
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 10, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
mattyclarkson added a commit to mattyclarkson/rules_go that referenced this issue Apr 25, 2024
The community has agreed[1] that "pure" builds of Go code should
use the `purego` build tag.

This change adds that de-facto tag.

The minimum version of Gazelle is bumped to `0.36.0` which includes a patch[2]
for ignoring the `purego` tag so that the source flies are passed to the
compiler for conditional compilation decisions.

[1]: golang/go#23172
[2]: bazelbuild/bazel-gazelle#1767
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests