code review 145390043: bufio: fix handling of empty tokens at end of line/file

bufio: fix handling of empty tokens at end of line/file
Fixes issue 8672.

[r, 2014-09-25 18:37:58]

Hello golang-codereviews@googlegroups.com,

I'd like you to review this change to
https://code.google.com/p/go

[rsc, 2014-09-25 19:36:42]

LGTM

https://codereview.appspot.com/145390043/diff/1/src/bufio/scan_test.go
File src/bufio/scan_test.go (right):

https://codereview.appspot.com/145390043/diff/1/src/bufio/scan_test.go#newcod...
src/bufio/scan_test.go:425: 
This test is way overkill. I spent a long time trying to understand it yesterday
in the playground. Here is a simpler test.

http://play.golang.org/p/3kaIKGj-Nb

func commaSplit(data []byte, atEOF bool) (advance int, token []byte, err error)
{
	for i := 0; i < len(data); i++ {
		if data[i] == ',' {
			return i+1, data[:i], nil
		}
	}
	if !atEOF {
		return 0, nil, nil
	}
	return 0, data, nil
}

func TestEmptyTokens(t *testing.T) {
	r := NewScanner(strings.NewReader("1,2,3,"))
	r.Split(commaSplit)
	var out []string
	for r.Scan() {
		out = append(out, r.Text())
	}
	want := []string{"1", "2", "3", ""}
	if !reflect.DeepEqual(out, want) {
		t.Errorf("scanned  %q, want %q", out, want)
	}
}

[r, 2014-09-25 19:45:24]

*** Submitted as https://code.google.com/p/go/source/detail?r=b35c4cbc07a2 ***

bufio: fix handling of empty tokens at end of line/file
Fixes issue 8672.

LGTM=rsc
R=golang-codereviews, rsc
CC=golang-codereviews
https://codereview.appspot.com/145390043