cmd/compile: allow the use of go:wasmimport globally #59149

johanbrandhorst · 2023-03-20T14:53:01Z

Background

#38248 defined a new compiler directive, go:wasmimport, for declaring a dependency on a function provided by the runtime environment. It also defined a restriction on the use of this compiler directive, such that it cannot be used in code outside of the runtime and syscall/js standard library packages. #58141 relaxed this restriction to allow the use of the directive in the syscall package to allow accessing WASI defined syscall APIs necessary to support WASI.

The go:wasmimport directive is defined as follows:

//go:wasmimport importmodule importname

Where importmodule is the Wasm module of the function and importname is the name of the function within the module. The effect of this compiler directive is that any Go function calls of the function will be replaced with a call instruction to the function identified by the module and function name during compilation. Parameters and return values to the Go function are translated to Wasm according to the following table:

Go value	Wasm value
int32, uint32	i32
int64, uint64	i64
float32	f32
float64	f64
unsafe.Pointer	i32

Any other parameter types are disallowed by the compiler.

For more information see the ssagen compiler package.

Proposal

We propose that the package restriction on the use of the go:wasmimport directive be removed entirely.

Rationale

Much of the discussion around the WASI proposal (#58141) revolved around the question of how and when the WASI target should move between syscall APIs (i.e. from depending on the wasm_snapshot_preview1 API to the wasm_snapshot_preview2 API and beyond). A major reason for this anxiety is because of the conflict between the desire for stability and the desire for access to newer runtime features afforded by newer syscall APIs.

One way to help avoid this conflict would be to provide access to newer APIs through purpose built third-party packages (e.g. golang.org/x/wasi/preview2/net) implementing various syscall level APIs through the use of go:wasmimport and exposing Go native types implementing interfaces such as net.Listener and net.Conn. Users who want to opt-in to new functionality can import one of these packages with the understanding that it may be unstable. Users who prefer to wait to use this functionality until it’s in the standard library can stay on the older API version. We’re exploring adding functionality to wit-bindgen to generate Go files with go:wasmimport directives and function signatures.

In addition to this, it would also allow users or vendors to create their own packages around vendor specific runtime extensions (e.g. WasmEdge defines its own sockets API, Fermyon defines a custom HTTP handler API) and expose them as easily consumable Go modules. This would help make Go more flexible and useful in different Wasm runtime environments.

Discussion

Rationale for the existing restriction

The existing restriction was added as a last minute addition to the original proposal. The primary rationale for this seems to have been an anxiety around exposing this widely without having had any experience of using it. If changes had to be made, it could break existing users.

We believe this concern should be smaller now that we have experience of using the directive while implementing go:wasmimport and working on implementing the WASI target.

Regardless, we do want to reserve the right to make breaking changes to this directive until such a point as the wasm architecture itself is declared stable. This is in line with the reservations made in #38248.

Effect on js/wasm

The js/wasm target has similar functionality exposed via the use of the syscall/js standard library package. The existing standard library interfaces will remain the preferred way of accessing functions defined on the JavaScript host. The primary use case within the js/wasm target would be as a means of accessing functions defined by other Wasm modules loaded into the same environment.

A note on wasm module names

The Wasm Core 1.0 spec defines a name as used in module and function names as a sequence of UTF-8 codepoints. The current definition of the go:wasmimport directive uses a space to separate the module name from the function name, making it impossible to specify a module or function name with a space in it. This proposal does not suggest changing the definition to support this use case.

Other language implementations

TinyGo

TinyGo uses the export pragma both for exporting to and importing methods from the environment. It also allows the user to specify the module to import from using the //go:wasm-module pragma. An example of this in the wild is the Fastly Compute@Edge SDK:

//go:wasm-module fastly_abi
//export init
//go:noescape
func fastlyABIInit(abiVersion prim.U64) FastlyStatus

TinyGo similarly restricts the types supported in parameters though I have struggled to find documentation for this exact limitation.

Rust

Rust uses the extern "C" raw interface pattern to define Wasm host imports. This is similar to our proposal in that it allows the user to specify the module and function name.

#[link(wasm_import_module = "the-wasm-import-module")]
extern "C" {
    // imports the name `foo` from `the-wasm-import-module`
    fn foo();

    // functions can have integer/float arguments/return values
    fn translate(a: i32) -> f32;

    // Note that the ABI of Rust and wasm is somewhat in flux, so while this
    // works, it's recommended to rely on raw integer/float values where
    // possible.
    fn translate_fancy(my_struct: MyStruct) -> u32;

    // you can also explicitly specify the name to import, this imports `bar`
    // instead of `baz` from `the-wasm-import-module`.
    #[link_name = "bar"]
    fn baz();
}

Rust supports using rich types in arguments but does not provide guarantees of the stability and encourages the use of integer/float values only.

Future work

Richer type support for imported functions

The current implementation of go:wasmimport supports only functions with integer, floating point or pointer parameters and a single return value. In the future, we would like to support types such as strings and structs and multiple return values. This would require defining a more elaborate calling convention for the imported functions than the one currently used.

The current mechanism follows the same semantics in use by other Wasm compilers such as Clang, which effectively treat Wasm the same as C, requiring the caller and callee to share the same ABI to be able to decode values in memory.

A 32 bit Wasm port

During the discussion of #59156, it became evident that many of the UX issues with the directive would disappear if the Wasm port was a 32 bit port, since all existing Wasm runtimes use 32 bit pointers, and the mismatch between the memory layout of Go and Wasm would disappear. There are still some concerns around whether the Go struct layout may change in the future, which would prevent us from guaranteeing direct memory mapping.

In the case of a future 32 bit Wasm port, the restriction on types allowed in the go:wasmimport parameters would be relaxed in a way that doesn't risk introducing memory corruption, but provides significant UX improvements to the compiler directive. The current wasm architecture, which is a 64 bit port, will likely retain the parameter limitations for the forseeable future.

64 bit Wasm runtimes

In the future, some runtimes may add support for 64 bit pointers. In such a runtime, the ABI translation rules for the use of the unsafe.Pointer type may change. We would evaluate the effects of this when/if adding support for 64 bit pointers.

Authors

@johanbrandhorst, @Pryz, @evanphx, @achille-roussel

The text was updated successfully, but these errors were encountered:

cherrymui · 2023-03-20T16:47:28Z

Pointer i32

Currently the implementation uses 32-bit linear memory. But that might change in the future? Does this have any impact to this? Thanks.

evanphx · 2023-03-20T16:51:15Z

@cherrymui There is the 64bit memory proposal for web assembly, so in the future yes. One reason we used i32 here is to remain similar to what clang and friends use for pointers. We certainly could use i64 today to avoid future concerns though.

cherrymui · 2023-03-20T16:58:32Z

Thanks. Using i32 for now is probably fine. The concern is that if later we change to 64-bit address, will this break any user? Maybe it will not if the Wasm engine is backward compatible? Do you think there is any compatibility issue? If not, this is probably okay.

evanphx · 2023-03-20T17:00:27Z

@cherrymui Ah great question. Let me dig into the 64bit memory proposal and see what they suggest, since this would effect everyone that generates wasm if they all need to compile different 32 vs 64 bit versions.

johanbrandhorst · 2023-03-20T20:14:46Z

I've raised #59156 which should clarify some things around behavior of pointers when using go:wasmimport. This proposal should only go ahead once #59156 has been resolved.

johanbrandhorst · 2023-04-07T01:10:38Z

I've added a section to the proposal that disallows any parameter types other than those that easily map to Wasm types. This restriction could be relaxed in a hypothetical future wasm32 port, but will likely remain in the current port until we have a better idea of what a 64 bit Wasm runtime means. See #59156 for more details on this discussion, but I believe this unblocks further discussion in this proposal.

cherrymui · 2023-04-07T15:38:28Z

Thanks. SGTM. Maybe add a note that if Wasm moves to 64-bit pointers, the ABI translation rule for unsafe.Pointer might change?

A minor question: if the directive is seen when compiling to a non-Wasm platform, is it an error or just silently ignored?

johanbrandhorst · 2023-04-08T02:31:41Z

Thank you, I updated the proposal with a section on future 64 bit Wasm runtimes.

The current behavior is to cause a compile time error when using the directive on a non-Wasm platform, and the current plan is to keep that behavior.

mathetake · 2023-04-10T11:02:26Z

One thing to note is that there will be some users who tries to put go:wasmimport on global variables from my experience with Ziglang dev ziglang/zig#4866 since import in Wasm can be applied to "wasm globals" and others as well, not only functions.

Support for Importing global variables will be far more complex than just functions, and standard compilers usually disable it AFAIK. So maybe we might want to add safeguard on this.

mathetake · 2023-04-10T11:07:54Z

Also, two things to consider:

do we want to raise an error on incompatible duplicated go:wasmimport directives at linktime? (somehow Wasm spec allows it, but in reality, such binaries are not loadable/usable).
wasm spec allows empty names both on module and symbol names. do we want to support them with go:wasmimport? I guess in practice no one would request, but it's good to be aware of what's possible in spec vs in Go runtime.

cherrymui · 2023-04-10T14:55:40Z

Support for Importing global variables will be far more complex than just functions, and standard compilers usually disable it AFAIK. So maybe we might want to add safeguard on this.

Yeah, if we support go:wasmimport only on functions, we should emit an error if the directive is applied to a variable.

do we want to raise an error on incompatible duplicated go:wasmimport directives at linktime? (somehow Wasm spec allows it, but in reality, such binaries are not loadable/usable).

Either way it is fine for me. If at link time we don't naturally check for duplications when generating the table, I think it is fine not to. Does other linker check this?

wasm spec allows empty names both on module and symbol names. do we want to support them with go:wasmimport? I guess in practice no one would request, but it's good to be aware of what's possible in spec vs in Go runtime.

I guess that will require changing the syntax of the directive? As noted above, we don't support space in module names, so I guess it is okay to leave empty name out as well. If there is a need in the future we can extended the syntax.

Empty names seem very confusing, though. How does the other side provide the symbol?

johanbrandhorst · 2023-04-10T17:21:17Z

do we want to raise an error on incompatible duplicated go:wasmimport directives at linktime? (somehow Wasm spec allows it, but in reality, such binaries are not loadable/usable).

Hm, theoretically, you could have libraries that import the same host function independently, right? Could you clarify what sort of problem this might cause?

wasm spec allows empty names both on module and symbol names. do we want to support them with go:wasmimport? I guess in practice no one would request, but it's good to be aware of what's possible in spec vs in Go runtime.

No, we don't want to allow this, and the current definition makes it impossible. Thanks for raising this!

mathetake · 2023-04-11T01:15:15Z

Hm, theoretically, you could have libraries that import the same host function independently, right? Could you clarify what sort of problem this might cause?

well, what if two different packages import the functions with different signatures but with the same import names? E.g.

package foo

//go:wasmimport env hostfn
func hostfn(uint32)

package bar

//go:wasmimport env hostfn
func hostfn(float32)

if we have objects like ^^, then the resulted wasm binary would be like:

(module
  (type (;0;) (func (param i32)))
  (type (;1;) (func (param f32)))
  (import "env" "hostfn" (func (;0;) (type 0)))
  (import "env" "hostfn" (func (;1;) (type 1)))
)

which is somehow valid in the sense of Wasm spec, but in reality the binary is unusable. So my question is whether or not we want to raise an error when linking such objects. But I guess as @cherrymui mentioned, we don't need to do anything.

edited: seems like clang/wasm-ld doesn't emit an error on this.

mathetake · 2023-04-11T01:22:23Z

Empty names seem very confusing, though. How does the other side provide the symbol?

in Wasm spec, names are arbitrary length utf-8 strings, so technically the hosts can define an empty module instance (zero length utf-8), and export the empty name entries (e.g. function).

agreed it's better to just rule out the empty names for now as it seems to require syntax changes as opposed to the practical use cases.

rsc · 2023-04-12T20:33:04Z

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

achille-roussel · 2023-04-15T19:36:04Z

@mathetake

well, what if two different packages import the functions with different signatures but with the same import names?

Since there will be a single signature eventually available at runtime, the program will likely fail at instantiation when resolving imports.

The compiler could ensure that all uses of go:wasmimport are consistent, but then they could also be all invalid or incompatible with the host functions available at runtime. I'm not arguing that we shouldn't add this type of validation, just reflecting on the limits of what can be verified at compile time.

Thanks for raising this point!

rsc · 2023-04-19T17:15:31Z

For the example with conflicting signatures, isn't one correct and one wrong? And if so, doesn't the wrong one cause a problem at runtime by itself?

Who are the maintainers of the wasm code that should weigh in? @cherrymui? @neelance? Anyone else?

cherrymui · 2023-04-19T17:22:15Z

As commented above #59149 (comment) I'm fine either way. I think we don't need to check if other toolchains don't. Yeah, I'd think it will result in some kind of run time (or instantiation time) error.

eliben · 2023-04-20T16:14:04Z

Agree that it's ok to defer this error checking to the WASM runtime if other toolchains do the same. As #59149 (comment) says, we can only check a subset of issues at compile time anyway. Runtime mismatches between imported symbols and actual host signatures are likely a much common problem, and this is something the compiler can't do anything about.

gopherbot · 2023-04-26T15:26:37Z

Change https://go.dev/cl/489255 mentions this issue: cmd/compile: remove go:wasmimport restriction

rsc · 2023-05-03T20:05:51Z

Based on the discussion above, this proposal seems like a likely accept.
— rsc for the proposal review group

rsc · 2023-05-10T21:56:07Z

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— rsc for the proposal review group

This is for compatibility with upstream Go. See golang/go#59149 for more context.

johanbrandhorst added Proposal arch-wasm WebAssembly issues labels Mar 20, 2023

gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 20, 2023

johanbrandhorst mentioned this issue Mar 20, 2023

cmd/compile: go:wasmimport allows pointers to pointers #59156

Closed

ianlancetaylor changed the title ~~cmd/compile: allow the use of go:wasmimport globally~~ proposal: cmd/compile: allow the use of go:wasmimport globally Mar 20, 2023

gopherbot added this to the Proposal milestone Mar 20, 2023

rsc added the Proposal-FinalCommentPeriod label May 3, 2023

rsc changed the title ~~proposal: cmd/compile: allow the use of go:wasmimport globally~~ cmd/compile: allow the use of go:wasmimport globally May 10, 2023

rsc modified the milestones: Proposal, Backlog May 10, 2023

rsc added Proposal-Accepted and removed Proposal-FinalCommentPeriod labels May 10, 2023

gopherbot closed this as completed in aff2f4a May 11, 2023

aykevl added a commit to tinygo-org/tinygo that referenced this issue May 19, 2023

compiler: disallow most types in //go:wasmimport

b2f6694

This is for compatibility with upstream Go. See golang/go#59149 for more context.

aykevl added a commit to tinygo-org/tinygo that referenced this issue May 19, 2023

compiler: disallow most types in //go:wasmimport

fadcff4

This is for compatibility with upstream Go. See golang/go#59149 for more context.

deadprogram mentioned this issue May 19, 2023

Restrict //go:wasmimport to a limited number of types tinygo-org/tinygo#3734

Merged

aykevl added a commit to tinygo-org/tinygo that referenced this issue May 19, 2023

compiler: disallow most types in //go:wasmimport

d77bd11

This is for compatibility with upstream Go. See golang/go#59149 for more context.

aykevl added a commit to tinygo-org/tinygo that referenced this issue May 19, 2023

compiler: disallow most types in //go:wasmimport

5885fce

This is for compatibility with upstream Go. See golang/go#59149 for more context.

deadprogram pushed a commit to tinygo-org/tinygo that referenced this issue May 20, 2023

compiler: disallow most types in //go:wasmimport

b08ff17

This is for compatibility with upstream Go. See golang/go#59149 for more context.

dmitshur modified the milestones: Backlog, Go1.21 Jun 4, 2023

LiuJiazheng pushed a commit to LiuJiazheng/tinygo that referenced this issue Aug 20, 2023

compiler: disallow most types in //go:wasmimport

2d0ce5f

This is for compatibility with upstream Go. See golang/go#59149 for more context.

deadprogram pushed a commit to tinygo-org/tinygo that referenced this issue Sep 17, 2023

compiler: disallow most types in //go:wasmimport

79a8291

This is for compatibility with upstream Go. See golang/go#59149 for more context.

johanbrandhorst mentioned this issue Sep 20, 2023

cmd/compile: create GOARCH=wasm32 #63131

Open

seankhliao mentioned this issue Jan 5, 2024

cmd/compile: docs for go:wasmimport #64972

Closed

orsinium mentioned this issue Mar 3, 2024

wasm: Allow multi-value return in wasmimport tinygo-org/tinygo#4171

Closed

johanbrandhorst mentioned this issue Apr 23, 2024

proposal: cmd/compile: relax wasm32 function import signature type constraints #66984

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/compile: allow the use of go:wasmimport globally #59149

cmd/compile: allow the use of go:wasmimport globally #59149

johanbrandhorst commented Mar 20, 2023 •

edited

cherrymui commented Mar 20, 2023

evanphx commented Mar 20, 2023

cherrymui commented Mar 20, 2023

evanphx commented Mar 20, 2023

johanbrandhorst commented Mar 20, 2023

johanbrandhorst commented Apr 7, 2023

cherrymui commented Apr 7, 2023

johanbrandhorst commented Apr 8, 2023

mathetake commented Apr 10, 2023

mathetake commented Apr 10, 2023 •

edited

cherrymui commented Apr 10, 2023

johanbrandhorst commented Apr 10, 2023

mathetake commented Apr 11, 2023 •

edited

mathetake commented Apr 11, 2023

rsc commented Apr 12, 2023

achille-roussel commented Apr 15, 2023 •

edited

rsc commented Apr 19, 2023

cherrymui commented Apr 19, 2023

eliben commented Apr 20, 2023

gopherbot commented Apr 26, 2023

rsc commented May 3, 2023

rsc commented May 10, 2023

cmd/compile: allow the use of go:wasmimport globally #59149

cmd/compile: allow the use of go:wasmimport globally #59149

Comments

johanbrandhorst commented Mar 20, 2023 • edited

Background

Proposal

Rationale

Discussion

Rationale for the existing restriction

Effect on js/wasm

A note on wasm module names

Other language implementations

TinyGo

Rust

Future work

Richer type support for imported functions

A 32 bit Wasm port

64 bit Wasm runtimes

Authors

cherrymui commented Mar 20, 2023

evanphx commented Mar 20, 2023

cherrymui commented Mar 20, 2023

evanphx commented Mar 20, 2023

johanbrandhorst commented Mar 20, 2023

johanbrandhorst commented Apr 7, 2023

cherrymui commented Apr 7, 2023

johanbrandhorst commented Apr 8, 2023

mathetake commented Apr 10, 2023

mathetake commented Apr 10, 2023 • edited

cherrymui commented Apr 10, 2023

johanbrandhorst commented Apr 10, 2023

mathetake commented Apr 11, 2023 • edited

mathetake commented Apr 11, 2023

rsc commented Apr 12, 2023

achille-roussel commented Apr 15, 2023 • edited

rsc commented Apr 19, 2023

cherrymui commented Apr 19, 2023

eliben commented Apr 20, 2023

gopherbot commented Apr 26, 2023

rsc commented May 3, 2023

rsc commented May 10, 2023

johanbrandhorst commented Mar 20, 2023 •

edited

mathetake commented Apr 10, 2023 •

edited

mathetake commented Apr 11, 2023 •

edited

achille-roussel commented Apr 15, 2023 •

edited