Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

syscall/js: request: want to convert slices from/to TypedArray directly #32402

Closed
hajimehoshi opened this issue Jun 3, 2019 · 9 comments
Closed
Labels
arch-wasm WebAssembly issues FeatureRequest FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.

Comments

@hajimehoshi
Copy link
Member

hajimehoshi commented Jun 3, 2019

As TypedArray no longer exists as of Go 1.13, there is no way to convert slices other than []byte from/to TypedArray objects other than Uint8Array. It is technically possible to do this by using encoding/binary, but I'm worried this conversion is not efficient.

As there are Web APIs that take such TypedArray objects (e.g., AudioBuffer.copyToChannel takes a Float32Array [1]), it would be very useful if syscall/js had such conversion functions.

CC @neelance

[1] https://developer.mozilla.org/en-US/docs/Web/API/AudioBuffer/copyToChannel

@hajimehoshi hajimehoshi changed the title syscall/js: request: want to convert slices to TypedArray directly syscall/js: request: want to convert slices from/to TypedArray directly Jun 3, 2019
@agnivade agnivade added arch-wasm WebAssembly issues FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jun 3, 2019
@dennwc
Copy link
Contributor

dennwc commented Jun 4, 2019

A possible workaround, for now, is to cast slices with unsafe and reflect.SliceHeader.

@hajimehoshi
Copy link
Member Author

@dennwc Thanks. Now I've succeeded to do what I wanted:

func sliceToByteSlice(s interface{}) []byte {
        switch s := s.(type) {
        case []int8:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                return *(*[]byte)(unsafe.Pointer(h))
        case []int16:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 2
                h.Cap *= 2
                return *(*[]byte)(unsafe.Pointer(h))
        case []int32:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 4
                h.Cap *= 4
                return *(*[]byte)(unsafe.Pointer(h))
        case []int64:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 8
                h.Cap *= 8
                return *(*[]byte)(unsafe.Pointer(h))
        case []uint8:
                return s
        case []uint16:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 2
                h.Cap *= 2
                return *(*[]byte)(unsafe.Pointer(h))
        case []uint32:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 4
                h.Cap *= 4
                return *(*[]byte)(unsafe.Pointer(h))
        case []uint64:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 8
                h.Cap *= 8
                return *(*[]byte)(unsafe.Pointer(h))
        case []float32:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 4
                h.Cap *= 4
                return *(*[]byte)(unsafe.Pointer(h))
        case []float64:
                h := (*reflect.SliceHeader)(unsafe.Pointer(&s))
                h.Len *= 8
                h.Cap *= 8
                return *(*[]byte)(unsafe.Pointer(h))
        default:
                panic(fmt.Sprintf("jsutil: unexpected value at sliceToBytesSlice: %T", s))
        }
}

func SliceToTypedArray(s interface{}) js.Value {
        switch s := s.(type) {
        case []int8:
                a := js.Global().Get("Uint8Array").New(len(s))
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Int8Array").New(buf, a.Get("byteOffset"), a.Get("byteLength"))
        case []int16:
                a := js.Global().Get("Uint8Array").New(len(s) * 2)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Int16Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/2)
        case []int32:
                a := js.Global().Get("Uint8Array").New(len(s) * 4)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Int32Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/4)
        case []uint8:
                a := js.Global().Get("Uint8Array").New(len(s))
                js.CopyBytesToJS(a, s)
                runtime.KeepAlive(s)
                return a
        case []uint16:
                a := js.Global().Get("Uint8Array").New(len(s) * 2)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Uint16Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/2)
        case []uint32:
                a := js.Global().Get("Uint8Array").New(len(s) * 4)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Uint32Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/4)
        case []float32:
                a := js.Global().Get("Uint8Array").New(len(s) * 4)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Float32Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/4)
        case []float64:
                a := js.Global().Get("Uint8Array").New(len(s) * 8)
                js.CopyBytesToJS(a, sliceToByteSlice(s))
                runtime.KeepAlive(s)
                buf := a.Get("buffer")
                return js.Global().Get("Float64Array").New(buf, a.Get("byteOffset"), a.Get("byteLength").Int()/8)
        default:
                panic(fmt.Sprintf("jsutil: unexpected value at SliceToTypedArray: %T", s))
        }
}

@johanbrandhorst
Copy link
Member

@hajimehoshi Is this available as a library somewhere?

@hajimehoshi
Copy link
Member Author

No so far. Ebiten has an internal package though: https://github.com/hajimehoshi/ebiten/tree/master/internal/jsutil

justinclift added a commit to justinclift/wasm-stl-viewer that referenced this issue Nov 6, 2019
Directly copied @hajimehoshi's workaround code from:

  golang/go#32402 (comment)

No idea if it'll work with TinyGo wasm yet, as it uses reflect.
Bluebugs pushed a commit to Bluebugs/gl that referenced this issue Feb 11, 2020
Bluebugs pushed a commit to Bluebugs/gl that referenced this issue Feb 12, 2020
Using information found in comment in golang/go#32402
@finnbear
Copy link

finnbear commented May 24, 2020

I resorted to patching wasm_exec.js with:

const loadFloatSlice = (addr) => {
	const array = getInt64(addr + 0);
	const len = getInt64(addr + 8);
	return new Float32Array(this._inst.exports.mem.buffer, array, len);
}
...
"syscall/js.copyBytesToJS": (sp) => {
	const dst = loadValue(sp + 8);
	let src;
	if (dst instanceof Uint8Array || dst instanceof Uint8ClampedArray) {
		src = loadSlice(sp + 16);
	} else if (dst instanceof Float32Array) {
		src = loadFloatSlice(sp + 16)
	} else {
		this.mem.setUint8(sp + 48, 0);
		return;
	}
	const toCopy = src.subarray(0, dst.length);
	dst.set(toCopy);
	setInt64(sp + 40, toCopy.length);
	this.mem.setUint8(sp + 48, 1);
},

These methods get called quite often in my WebGL game and I can't afford indirection or extra syscalls used here:
https://github.com/hajimehoshi/ebiten/blob/982a68e5a21f3b2dff494132b34639084fd3e9ac/internal/jsutil/go113_js.go#L104

Looking for something similar to be included in the standard syscall/js library (supporting all other typed arrays too, not just Float32 which I am currently restricting myself to). An API like CopyFloat32sToJS would be perfect!

unitoftime pushed a commit to unitoftime/gl that referenced this issue Jun 26, 2020
Using information found in comment in golang/go#32402
@neelance
Copy link
Member

@finnbear For performance critical code you probably want to reuse a single buffer anyways. So first create a Uint8Array and Float32Array that share the same buffer and then use copyBytesToJS with sliceToByteSlice (see above) multiple times. This should have no additional overhead.

@neelance
Copy link
Member

neelance commented Nov 5, 2020

@hajimehoshi @finnbear Are you both happy with what is already possible? Shall we close this issue?

@finnbear
Copy link

finnbear commented Nov 6, 2020

@neelance I'm fine with CopyBytesToJS as-is, mostly because the option to use multiple typed arrays referencing the same buffer is available. I highly suggest you document that, although I'm still using the hack I showed above because my application commonly copies float32s and bytes. I'm fine with this issue being closed 👍

@hajimehoshi
Copy link
Member Author

I'm fine with closing this issue.

@neelance neelance closed this as completed Nov 6, 2020
misak113 added a commit to misak113/webgl that referenced this issue Mar 9, 2021
* where TypedArrayOf was removed & replaced with other techniques
* the conversion was inspired & copied from issue golang/go#32402
misak113 added a commit to misak113/webgl that referenced this issue Mar 9, 2021
* where TypedArrayOf was removed & replaced with other techniques
* the conversion was inspired & copied from issue golang/go#32402
@golang golang locked and limited conversation to collaborators Nov 6, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
arch-wasm WebAssembly issues FeatureRequest FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants