Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/binary: add Append to encode values into a provided buffer #60023

Open
lmb opened this issue May 6, 2023 · 3 comments
Open
Labels
Milestone

Comments

@lmb
Copy link
Contributor

lmb commented May 6, 2023

I'd like to propose adding a function with the following signature:

// Append the binary representation of data to buf.
//
// buf may be nil, in which case a new buffer will be allocated. See [Write] on
// which data are acceptable.
//
// Returns the (possibly extended) buffer containing data or an error.
func Append(buf []byte, order AppendByteOrder, data any) ([]byte, error)

This is useful when repeatedly encoding the same kind of value multiple times into a larger buffer and is a natural extension to #50601. A related proposal wants to add similar functions to other packages in encoding: #53693.

Together with #53685 it becomes possible to implement a version of binary.Write that doesn't allocate when using common io.Writer. See my comment for writeBuffer(). Roughly (untested):

func write(w io.Writer, order binary.AppendByteOrder, data any) error {
    buf := writeBuffer(w, binary.Size(data))
    binary.Append(buf[:0], order, data)
    _, err := w.Write(buf)
   return err
}

If the CLs to avoid escaping in reflect APIs lands, Append would allow encoding with zero allocations.

I think it might also allow encoding into stack allocated slices, provided the compiler is (or becomes) smart enough:

buf := binary.Append(make([]byte, 0, 128), binary.LittleEndian, v)
@lmb lmb added the Proposal label May 6, 2023
@gopherbot gopherbot added this to the Proposal milestone May 6, 2023
@ianlancetaylor
Copy link
Contributor

CC @dsnet

@dsnet
Copy link
Member

dsnet commented May 9, 2023

SGTM. Write is the last API in binary that lacks an append-like equivalent.

@lmb
Copy link
Contributor Author

lmb commented May 19, 2023

I've been playing with the implementation a bit, and I'd like to extend my proposal to cover an equivalent function for Read.

// Decode data from buf according to the given byte order.
//
// Returns the number of bytes read from the beginning of buf or io.ErrUnexpectedEOF.
func Decode(buf []byte, order ByteOrder, data any) (int, error)

The rationale is similar to Append: using this new function makes it possible to implement zero-allocation decode for common io.Reader.

I struggled to come up with a good name for the function, what is the inverse of Append? Renaming it to Encode seems clearer to me, and follows existing unexported types. The proposal would become:

func Decode(buf []byte, order ByteOrder, data any) (int, error) // aka Read
func Encode(buf []byte, order AppendByteOrder, data any) ([]byte, error) // aka Write

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

4 participants