Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/binary: add NativeEndian #35398

Closed
josharian opened this issue Nov 6, 2019 · 3 comments
Closed

proposal: encoding/binary: add NativeEndian #35398

josharian opened this issue Nov 6, 2019 · 3 comments

Comments

@josharian
Copy link
Contributor

This was previously discussed at https://groups.google.com/forum/#!topic/golang-nuts/3GEzwKfRRQw.

The rationale given there (by multiple people, these words by Paul Borman) was:

This comes up when talking with the kernel and also with networking between processes on the same machine (or like machines) that might be using a traditionally defined structure rather than some dynamic structure.

Rob replied:

[...] almost every time I hear someone asking what the native byte order is, it's a mistake.

Russ wrote:

Put [var hbo = binary.LittleEndian] in a file named byteorder_amd64.go and it stops being a hack. It need not be in the standard library.

I'd like to add another rationale for adding it: encoding/binary can act as a poor man's vectorization. With a bit of care, you can read eight bytes into a uint64 with it, operate on the uint64, and write back eight bytes, and have the code emitted be just a MOVQ, XORQ, MOVQ. It will also be correct and safe across all architectures and alignments, because it is pure Go. See #31586 (comment) and #35381.

With neither implicit nor explicit vectorization anywhere on the horizon, this is pretty useful. However, this is most important in performance-sensitive code like crypto, where using a non-native endianness incurs a non-trivial penalty (best case a bswap).

Russ suggested using build tags. There are a few downsides to this. First, the obvious implementation (var nativeEndian = binary.LittleEndian) introduces a memory load into the hot path, since the compiler can't prove that nativeEndian is never modified and changes a direct call to an indirect one (albeit a highly predictable one). Second, the burden of maintaining a list of all architectures organized by endianness falls on all clients, rather than one centralized location.

Another option would be to add an endianness constant somewhere (package runtime? math/bits? encoding/binary?).

@rsc
Copy link
Contributor

rsc commented Nov 6, 2019

First, the obvious implementation (var nativeEndian = binary.LittleEndian) introduces a memory load into the hot path, since the compiler can't prove that nativeEndian is never modified and changes a direct call to an indirect one (albeit a highly predictable one).

In this case nativeEndian will have type binary.littleEndian. Pretty sure the compiler can prove that type never changes. :-)

I still agree with Rob that NativeEndian is almost always a mistake. Let's solve vectorization a better way than misleading API hacks in encoding/binary that we will never be able to remove.

@ddevault
Copy link

ddevault commented Dec 8, 2019

I have a different use-case, which cannot reasonably be solved in a more clever way by attacking the high-level use-cases. I am writing a Wayland protocol implementation for Go, which communicates with third-party client/server implementations over a Unix socket. The byte order for this protocol is defined as host endianness.

@josharian
Copy link
Contributor Author

I needed this yet again, so I made this: https://github.com/josharian/native

@golang golang locked and limited conversation to collaborators Aug 17, 2021
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants