encoding: provide append-like variants #53693
Comments
|
As far as names go, we could also consider EncodeAppend and DecodeAppend, even if they aren't as consistent with the usual "verb then object" method names like MarshalJSON or AppendUint. |
|
This proposal has been added to the active column of the proposals project |
|
Based on the discussion above, this proposal seems like a likely accept. |
|
No change in consensus, so accepted. 🎉 |
rsc
changed the title
|
Change https://go.dev/cl/504884 mentions this issue: |
|
Change https://go.dev/cl/520755 mentions this issue: |
For #53693 Change-Id: I6a428a4a10a2e2efa03296f539e190f0743c1f46 Reviewed-on: https://go-review.googlesource.com/c/go/+/520755 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Run-TryBot: Andy Pan <panjf2000@gmail.com>
For golang#53693 Change-Id: I6a428a4a10a2e2efa03296f539e190f0743c1f46 Reviewed-on: https://go-review.googlesource.com/c/go/+/520755 Reviewed-by: Dmitri Shuralyov <dmitshur@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> Reviewed-by: Ian Lance Taylor <iant@google.com> TryBot-Result: Gopher Robot <gobot@golang.org> Auto-Submit: Dmitri Shuralyov <dmitshur@golang.org> Run-TryBot: Ian Lance Taylor <iant@google.com> Auto-Submit: Ian Lance Taylor <iant@google.com> Run-TryBot: Andy Pan <panjf2000@gmail.com>
|
@dsnet: Thank you for adding those. I was also looking into how to append encoded and now I see it will be out in upcoming 1.22. Great! But I am still curious how this interacts with To me, the following works (using what is available with 1.21): v := []byte{'a', 'b'}
b := &bytes.Buffer{}
data := make([]byte, base64.StdEncoding.EncodedLen(len(v)))
base64.StdEncoding.Encode(data, v)
b.Write(data)But if I do: v := []byte{'a', 'b'}
b := &bytes.Buffer{}
b.Grow(base64.StdEncoding.EncodedLen(len(v)))
base64.StdEncoding.Encode(b.AvailableBuffer(), v)
fmt.Println(b.String())if fails with a panic. I must say I do not understand why based on discussion in #53685, it seems this should also work? |
|
OK, I made it work: v := []byte{'a', 'b'}
buf := &bytes.Buffer{}
l := base64.StdEncoding.EncodedLen(len(v))
buf.Grow(l)
b := buf.AvailableBuffer()[:l]
base64.StdEncoding.Encode(b, v)
buf.Write(b)
fmt.Println(buf.String())No I understand why |
|
Change https://go.dev/cl/547755 mentions this issue: |
For #53693. Change-Id: I360f5cb9caf5fa77267a100eebcc282955677abe Reviewed-on: https://go-review.googlesource.com/c/go/+/547755 Reviewed-by: Robert Griesemer <gri@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Auto-Submit: Robert Griesemer <gri@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> TryBot-Bypass: Robert Griesemer <gri@google.com>
For golang#53693. Change-Id: I360f5cb9caf5fa77267a100eebcc282955677abe Reviewed-on: https://go-review.googlesource.com/c/go/+/547755 Reviewed-by: Robert Griesemer <gri@google.com> Reviewed-by: Michael Knyszek <mknyszek@google.com> Auto-Submit: Robert Griesemer <gri@google.com> Reviewed-by: Joseph Tsai <joetsai@digital-static.net> TryBot-Bypass: Robert Griesemer <gri@google.com>
There's been a shift in recent years to provide
Append-like variants of APIs:as these are generally easier to work with than
Put-like APIs where you need to know the size of the buffer beforehand, prepare a buffer of the right size, and then call thePut-like operation.Such APIs are missing for the
encoding/hex,encoding/base32, andencoding/base64packages. I propose we add the following:This would simplify many callers of these packages, where the only reason to call the
Lenmethod is to obtain the size of the expected output buffer, do some manual memory management, and then call theEncodeorDecodeequivalent.For example, this logic in the
jsonpackage:go/src/encoding/json/encode.go
Lines 839 to 860 in c111091
could be simplified as:
This assumes we had some way to unify
bytes.Bufferwithappend-like APIs; see #53685 (comment).The resulting code is both simpler and more performant since we can append directly into the underlying
bytes.Bufferwithout going through an intermediateencodeState.scratchbuffer. Also, the caller doesn't have to do manual memory management checking whether the encoded output fits within theencodeState.scratch.The text was updated successfully, but these errors were encountered: