Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

net/textproto: swallows errors returned by underlying reader #53858

Open
kayrus opened this issue Jul 13, 2022 · 13 comments
Open

net/textproto: swallows errors returned by underlying reader #53858

kayrus opened this issue Jul 13, 2022 · 13 comments
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@kayrus
Copy link

kayrus commented Jul 13, 2022

What version of Go are you using (go version)?

$ go version
go version go1.18.3 linux/amd64

Does this issue reproduce with the latest release?

Yes.

What did you do?

See example: https://go.dev/play/p/Bn8H6p4QUf6

What did you expect to see?

I expect that bufio package react on underlying reader errors and stop the reading.

What did you see instead?

I see that bufio skips an error multiple times, and the example code causes OOM error becuase reader is rewinded every time:

here are cases when an error is not tracked:
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/net/textproto/reader.go;l=148;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/net/textproto/reader.go;l=496;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/net/textproto/reader.go;l=566;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/net/textproto/reader.go;l=571;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673

the error, returned by autoRewind is just ignored and set to nil:
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/bufio/bufio.go;l=156;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673
https://cs.opensource.google/go/go/+/refs/tags/go1.18.4:src/bufio/bufio.go;l=124;drc=27794c4d4a18c61d8c158d253421d72b5a6a8673

I know that it's something that I have to expect with autoRewind, but this behavior caused an unpleasant surprise.

see also:

@seankhliao
Copy link
Member

Your code never returns an error. There is nothing for bufio to report.
If you fix your code to actually return errors, it still violates the io.Reader contract:

a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF

Closing as there is nothing to do.

For questions please refer to https://github.com/golang/go/wiki/Questions

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Jul 13, 2022
@kayrus
Copy link
Author

kayrus commented Jul 13, 2022

Your code never returns an error. There is nothing for bufio to report.

@seankhliao it does, see line 29

@seankhliao
Copy link
Member

Your Read creates a fresh strings.Reader every time because its defined with a value receiver, and strings.Reader doesn't return an EOF until the second read.

@kayrus
Copy link
Author

kayrus commented Jul 14, 2022

@seankhliao I updated the example code: https://go.dev/play/p/XD0e4msIGqt
you can see that simple io.Copy works, simple io.Copy wrapped around bufio also works, but net/textproto read fails.
Yes, I was wrong with the issue topic, so I renamed it to net/textproto.

@kayrus kayrus changed the title bufio: swallows errors returned by underlying reader net/textproto: swallows errors returned by underlying reader Jul 14, 2022
@ianlancetaylor
Copy link
Contributor

Thanks, I do think there is a bug here. net/textproto.Reader.skipSpace assumes that it doesn't have to return the error, with a comment saying "Bufio will keep err until next read.", but that's not how the bufio package works. Once the bufio package has returned an error, it won't return it again.

@ianlancetaylor ianlancetaylor added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Jul 14, 2022
@ianlancetaylor ianlancetaylor added this to the Backlog milestone Jul 14, 2022
@cz2h
Copy link

cz2h commented Jul 18, 2022

@ianlancetaylor One quick fix I can think of is to add one more field in bufio's Reader struct to memorize if EOF is reached and let ReadLine() returns while EOF is reached before calling ReadSlice() for the following reasons.

In the case of using a non-autorewind reader, after reading the last line of message with bufio.ReadLine(), there will be one more call on bufio.ReadLine() and the EOF error pops up.

In the case of using autorewind reader, the bufio.ReaderLine() seems to always being able to read something without returning the error and the methods in textproto calling ReadLine() will always read something without receiving the EOF error, making the ReadMIMEHeader() non stop.

Please point out if there anything unclear or unthoughtful above, thanks

@ianlancetaylor
Copy link
Contributor

The bufio package is pretty low level and it would be much better if we didn't have to change it.

As far as I can see the bug here is in net/textproto, and that is where the fix should occur.

@thedadams
Copy link

Could I make an attempt at fixing this issue?

cz2h added a commit to cz2h/go that referenced this issue Jul 20, 2022
The existing implementation of ReadMIMEHeader() relies on the assumption that bufio will keep the EOF error. The for loop will not terminates until a EOF occurs or a len 0 buf is read. In the case where a rewind reader is use, bufio will always return a string and ignore the EOF error. Hence an variable is ued to indicate that EOF is reached.

A corresponding test cases is provided.

Fix golang#53858
@gopherbot
Copy link

Change https://go.dev/cl/418694 mentions this issue: net/textproto: add extra field to indicate whether EOF is reached

@gopherbot
Copy link

Change https://go.dev/cl/421554 mentions this issue: net/textproto: refactor return type of skipSpace function

@gopherbot
Copy link

Change https://go.dev/cl/544935 mentions this issue: net/textproto: improve error handling in skipSpace

@ianlancetaylor
Copy link
Contributor

Looking again at the original example, the Reader there looks like this:

func (r *autoRewind) Read(p []byte) (int, error) {
	if r.r == nil {
		r.r = strings.NewReader(r.buf)
	}

	n, err := r.r.Read(p)
	fmt.Printf("%d: %v\n", n, err)
	if err == io.EOF {
		// rewind
		r.r = strings.NewReader(r.buf)
	}

	return n, err
}

This is going to return io.EOF and then continue to return data. That violates the io.Reader contract, which says

When Read encounters an error or end-of-file condition after successfully reading n > 0 bytes, it returns the number of bytes read. It may return the (non-nil) error from the same call or return the error (and n == 0) from a subsequent call. An instance of this general case is that a Reader returning a non-zero number of bytes at the end of the input stream may return either err == EOF or err == nil. The next Read should return 0, EOF.

That is, after returning EOF, the Reader should then return 0, EOF. The code in this example does not do that.

So now I'm not sure whether there is a bug here at all.

@joeshin
Copy link

joeshin commented Feb 24, 2024

Thanks for looking this one more time @ianlancetaylor. Re-reading the thread again as well as your last comment, I see what @seankhliao was saying. I agree, the example implementation doesn't do that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

7 participants