CC @dsnet

I don’t see how this provides any greater functionality over the BinaryAppender interface… If the io.Writer is a bytes.Buffer, then the later is just wrapping a []byte internally, and thus suffers from the same exact reallocation issues as holding the byte slice yourself. And using a b = b[:0] to reset your byte slice, is not just vaguely the same, it’s the exact same first line of code in bytes.Buffer.Reset(), and if it didn’t need to maintain an io.Reader implementation, it wouldn’t need at all the second two lines of code.

Looking at the linked examples… I’m unsure how that code is bringing any advantage to GC pressure over just using BinaryAppender. Indeed, the ergo/lib.Buffer is throwing away just as many backing arrays on reallocations… the only difference is really that you’re keeping a side-band backing slice with a minimum size that you could have just already turned to the pool as soon as it was unused after a append call that would reallocate.

But I don’t want to get too in the weeds on that specific implementation, because my central consideration is what behavior does this provide that is not covered by good BinaryAppender use. I’m not saying that there’s zero benefit over BinaryAppender, but “reduced GC pressure” is not it, right?

@puellanivis thanks for your comment.

The main idea of BinaryWriter is to provide full control over the underlying buffer, which essentially gives you control over the pressure on the GC. That's basically what I do in my project (Ergo Framework).

With BinaryAppender, double reallocation is the usual case

With BinaryAppender, double reallocation is the usual case

But then the BinaryWriter would cause allocations on the caller side, right? BinaryWriter.WriteBinary(slice) would in most cases cause the slice to escape to the heap.

would in most cases cause the slice to escape to the heap.

escapes only if buffer > 2^16

const size = 1 << 16

func write(b []byte) error {
	b[0] = 1
	return nil
}

func Benchmark_LargeSize_Stack_EqualOrLess65535(b *testing.B) {
	for i := 0; i < b.N; i++ {
		// not escape to heap when size < 2^16
		dataLarge := make([]byte, size)
		write(dataLarge)
	}
}
func Benchmark_LargeSize_Heap_LargerThan65535(b *testing.B) {
	for i := 0; i < b.N; i++ {
		// escape to heap when size is 2^16 and greater
		dataLarge := make([]byte, size+1)
		write(dataLarge)
	}
}

With BinaryAppender, double reallocation is the usual case

I don’t believe I understand what you mean here. Can you demonstrate some code that would “double reallocate” here, that could not easily be solved ahead of the code with a b = slices.Grow(b, expectedAdditionalNecessaryLength)?

Like:

func (f *Foo) AppendBinary(b []byte) ([]byte, error) {
  b = slices.Grow(b, 42)
  b = append(b, []byte{ /* 32 bytes of data */ }...)
  if cond {
    return append(b, []byte{ /* 10 bytes of data */ }...), nil
  }
  return append(b, []byte{ /* 10 different bytes of data */ }...), nil
}

Like, equally in your example, one could use b := make([]byte, 0, size) and then pass that through the various BinaryAppenders, right? And it wouldn’t escape into the heap the same as your benchmem demonstrates? And it would have at most one allocations on the demonstrated Foo.AppendBinary() as well.

isn't this just the io.WriterTo interface?
https://pkg.go.dev/io#WriterTo

escapes only if buffer > 2^16

You are testing a completely different case, you have to call an interface with slice in a way that the call is not devirtualized.

escapes only if buffer > 2^16

You are testing a completely different case, you have to call an interface with slice in a way that the call is not devirtualized.

is it ok now or still a different case? :)

const size = 1 << 16

type A struct {
}

func (a *A) Write(b []byte) (int, error) {
	b[0] = 1
	return 0, nil
}

var (
	aaa = &A{}
)

func Benchmark_LargeSize_Stack_EqualOrLess65535(b *testing.B) {
	var writer io.Writer = aaa
	for i := 0; i < b.N; i++ {
		// not escape to heap when size < 2^16
		dataLarge := make([]byte, size)
		writer.Write(dataLarge)
	}
}
func Benchmark_LargeSize_Heap_LargerThan65535(b *testing.B) {
	var writer io.Writer = aaa
	for i := 0; i < b.N; i++ {
		// escape to heap when size is 2^16 and greater
		dataLarge := make([]byte, size+1)
		writer.Write(dataLarge)
	}
}

The global aaa must be of an iface type (so var aaa io.Writer = &A{}), since otherwise writer.Write is devirtualized directly to (*A).Write.

The global aaa must be of an iface type (so var aaa io.Writer = &A{}), since otherwise writer.Write is devirtualized directly to (*A).Write.

thanks, missed that

isn't this just the io.WriterTo interface? https://pkg.go.dev/io#WriterTo

:blink: :blink: It never occurred to me that one could totally use it like that? 🤦‍♀ Like, to write a non-data stream thing to an io.Writer.

:blink: :blink: It never occurred to me that one could totally use it like that? 🤦‍♀ Like, to write a non-data stream thing to an io.Writer.

Also this is a pattern that we already expose in few APIs like: https://pkg.go.dev/golang.org/x/tools/go/ssa#Function.WriteTo

:blink: :blink: It never occurred to me that one could totally use it like that? 🤦‍♀ Like, to write a non-data stream thing to an io.Writer.

Also this is a pattern that we already expose in few APIs like: https://pkg.go.dev/golang.org/x/tools/go/ssa#Function.WriteTo

Also, from http.Request, we have Write(w io.Writer) error which would match this proposed interface but for name. But it also writes in a textproto, right? 🤔

I think the interface might be a good idea in general… even if I’m not 100% sold on the “reduce GC pressure” argument.