Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bufio: the way to determine bufio.Reader size #21343

Closed
gobwas opened this issue Aug 8, 2017 · 19 comments
Closed

bufio: the way to determine bufio.Reader size #21343

gobwas opened this issue Aug 8, 2017 · 19 comments
Labels
FeatureRequest FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@gobwas
Copy link

gobwas commented Aug 8, 2017

Hello!

For reusing purposes, I need to determine the bufio.Reader underlying buffer size.
That is, for bufio.Writer there is bw.Available() method. Is there (or planning to be) something similar for bufio.Reader?

@mvdan
Copy link
Member

mvdan commented Aug 8, 2017

@gobwas could you give a small example of a program that would show the need for this? I can't picture why knowing the available (i.e. unused, not total) number of bytes in a buffer would help you reuse it.

@gobwas
Copy link
Author

gobwas commented Aug 8, 2017

@mvdan no, I want to get exactly the total number of bytes: len(br.buf).

var pool = map[int]*sync.Pool{
	16: &sync.Pool{},
	32: &sync.Pool{},
	// ...
}

func GetReader(r io.Reader, n int) (br *bufio.Reader) {
	// round n to power of two somehow.

	if p, ok := pool[n]; ok {
		br, _ = p.Get().(*bufio.Reader)
	}
	if br == nil {
		br = bufio.NewReaderSize(r, n)
	}
	return br
}

func PutReader(br *bufio.Reader) {
	n := ... // get the size of br's underlying buffer.
	if p, ok := pool[n]; ok {
		br.Reset(nil)
		p.Put(br)
	}
}

@gobwas
Copy link
Author

gobwas commented Aug 8, 2017

The only way I could get the size is this workaround:

type optimisticReader struct{}

func (optimisticReader) Read(p []byte) (int, error) {
	return len(p), nil
}

func readerSize(br *bufio.Reader) int {
	br.Reset(optimisticReader{})
	br.ReadByte()
	return br.Buffered() + 1
}

@mvdan
Copy link
Member

mvdan commented Aug 8, 2017

You said you need something in bufio.Reader that is like bufio.Writer.Available. My point was that Available reports the available number of bytes, not the total.

You can set the size when you create the buffer. Can you not keep track of it? Exporting fields and methods is always an option, but it always comes at the cost of extra complexity and maintenance work. The fact that I don't recall anyone (nor me) saying that they needed this before makes me wonder if the tradeoff would be worth it.

@bradfitz
Copy link
Contributor

bradfitz commented Aug 8, 2017

I'm pretty sure I had a TODO in some repo to expose the bufio.Reader size but grepping around, I can't find it. But this is also something I've wanted a few times now.

@bradfitz bradfitz added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Aug 8, 2017
@bradfitz bradfitz added this to the Go1.10 milestone Aug 8, 2017
@gobwas
Copy link
Author

gobwas commented Aug 8, 2017

Sorry for missing details in my first message, @mvdan. I meant bw.Reset(nil) followed by bw.Available(), not just bw.Available(). Sorry.

@rsc
Copy link
Contributor

rsc commented Aug 21, 2017

@mvdan asked above why you need this. I haven't seen anyone answer that.

@gobwas
Copy link
Author

gobwas commented Aug 22, 2017

@rsc I need this to reuse bufio.Reader instances by its size. Did you have a look at this comment?

@griesemer
Copy link
Contributor

@gobwas I think the request is not unreasonable, but before we add more to this API, is there any reason why b.Reset(nil) followed by b.Buffered() is not an option? In other words, since this is presumably about re-use of buffers (I guess to avoid problems with excessive allocation), is there a problem with resetting the buffer to get to the number? And if so, what is the problem? Finally, since you are maintaining the buffers anyway, couldn't you just keep track of the sizes as well (as @mvdan already asked)?

@gobwas
Copy link
Author

gobwas commented Aug 22, 2017

@griesemer what do you expect to get after b.Reset(nil); b.Buffered()? Keeping track of sizes leads to additional allocations or wrappers around bufio.Reader, that is undesirable.

@griesemer
Copy link
Contributor

@gobwas Never mind - b.Buffered() wouldn't work of course. But as a work-around, an additional wrapper would work, or would it not? (ignoring the undesirability for a moment).

@gobwas
Copy link
Author

gobwas commented Aug 22, 2017

@griesemer this workaround would work too without wrappers. But I want it to be a native method for bufio.Reader.

@rogpeppe
Copy link
Contributor

FWIW this works:

 r.Reset(nil)
 b, _ := r.Peek(0)
 bufferLength := cap(b)

but it does rely on Peek not setting the capacity of the returned buffer, which I'm not sure is covered by the compatibility guarantee.

@gobwas
Copy link
Author

gobwas commented Aug 23, 2017

@rogpeppe thanks =) Yes, this looks like much shorter but still workaround.

@gobwas
Copy link
Author

gobwas commented Oct 19, 2017

Ping )

@ianlancetaylor
Copy link
Contributor

Sorry, this isn't a case where pinging will make anything happen faster. We need to decide whether and how to expand the API. The issue is marked to be resolved for the 1.10 release, and we'll do our best.

@rsc
Copy link
Contributor

rsc commented Oct 30, 2017

This looks OK. I'll send a CL.

package  bufio
// Size returns the size of the underlying buffer in bytes.
func (r *Reader) Size() int
// Size returns the size of the underlying buffer in bytes.
func (w *Writer) Size() int

@rsc rsc added the NeedsFix The path to resolution is known, but the work has not been done. label Oct 30, 2017
@gopherbot gopherbot removed the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 30, 2017
@rsc rsc self-assigned this Oct 30, 2017
@gopherbot
Copy link

Change https://golang.org/cl/75150 mentions this issue: bufio: add Reader.Size and Writer.Size accessors

@bradfitz
Copy link
Contributor

bradfitz commented Nov 1, 2017

I needed a warm-up task so I sent off https://go-review.googlesource.com/#/c/go/+/75150

@golang golang locked and limited conversation to collaborators Nov 2, 2018
@rsc rsc removed their assignment Jun 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FeatureRequest FrozenDueToAge NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

No branches or pull requests

8 participants