Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ReadAllLimitSize to io/ioutil #28788

Closed
fwhezfwhez opened this issue Nov 14, 2018 · 5 comments
Closed

Add ReadAllLimitSize to io/ioutil #28788

fwhezfwhez opened this issue Nov 14, 2018 · 5 comments

Comments

@fwhezfwhez
Copy link

What do I do?

Add new function for io/ioutil package.

What does it work for?

Read a reader with a size limit.When using ioutil.ReadAll(), I'm afaid the reader contains too much to fullfill the memory.

What is it?

func ReadAllLimitSize(r io.Reader, maxSize int64) ([]byte, error) {
	// when maxSize <= 0, read all
	if maxSize <= 0 {
		return ioutil.ReadAll(r)
	}
	const (
		B  = 1
		KB = 1024
		MB = 1024 * 1024
		GB = 1024 * 1024 * 1024
	)
	result := make([]byte, 0, 512)
	step,n :=0,0
	buf := make([]byte,512)
	var er error
	for {
		step ,er =r.Read(buf)
		n += step
		if int64(n) >= maxSize {
			return nil, errors.New(fmt.Sprintf("read  %d,but limit %d", n, maxSize))
		}

		result = append(result, buf[:step]...)
		if er!=nil && er == io.EOF{
			break
		}
		if er !=nil {
			return nil,er
		}
	}
	return result, nil
}

Testcase

func TestReadAllLimitSize(t *testing.T) {
	buf,e :=ReadAllLimitSize(strings.NewReader("hello world"), 122)
	if e!=nil {
		fmt.Println(e.Error())
		t.Fail()
	}
	fmt.Println(string(buf))
}

// 5000000	       214 ns/op
func BenchmarkReadAllLimitSize(b *testing.B) {
	r := strings.NewReader("benchmark for readAllLimitSize")
	const maxSize = int64(100)
	var buf []byte
	var e error
	for i:=0;i<b.N;i++{
		buf,e =ReadAllLimitSize(r, maxSize)
		if e!=nil{
			b.Fatal(e.Error())
		}
	}
	fmt.Println(string(buf))
}
@agnivade
Copy link
Contributor

I believe you are looking for - https://golang.org/pkg/io/#LimitReader

According to your code, it should be ioutil.ReadAll((io.LimitReader(r, 100)).

I am going to close this as I see that the functionality is already present in the standard library. Adding another function which does the exact same thing will lead to unnecessary duplication. Please feel free to comment if you still want it, or had something else in mind.

Thank you.

@fwhezfwhez
Copy link
Author

3q

@fwhezfwhez
Copy link
Author

@agnivade hello, ioutil.ReadAll((io.LimitReader(r, 100)) only read the first 100 byte throwing no error.How can I make an error to warn the caller.

@mvdan
Copy link
Member

mvdan commented Nov 14, 2018

@fwhezfwhez for questions about Go, see https://golang.org/wiki/Questions.

@fwhezfwhez
Copy link
Author

well

@golang golang locked and limited conversation to collaborators Nov 15, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

4 participants