code review 169970043: bufio: don't loop generating empty tokens

bufio: don't loop generating empty tokens

The new rules for split functions mean that we are exposed
to the common bug of a function that loops forever at EOF.
Pick these off by shutting down the scanner if too many
consecutive empty tokens are delivered.

Fixes issue 9020.

[r, 2014-11-05 01:49:15]

Hello golang-codereviews@googlegroups.com,

I'd like you to review this change to
https://code.google.com/p/go

[rsc, 2014-11-05 01:59:05]

LGTM

Should it panic instead? 100 bogus tokens have been returned.

[adg1, 2014-11-05 02:00:09]

https://codereview.appspot.com/169970043/diff/1/src/bufio/scan.go
File src/bufio/scan.go (right):

https://codereview.appspot.com/169970043/diff/1/src/bufio/scan.go#newcode133
src/bufio/scan.go:133: if s.empties > 100 {
100 seems low
and this should certainly be documented
someone is going to scan a file with >100 blank lines and get really confused
when it stops scanning partway through

[bradfitz, 2014-11-05 02:07:09]

I hope one of you is wrong. Reading a file with bufio.Scanner, any number
of blank lines (millions!) should be possible.
 On Nov 4, 2014 11:00 PM, "adg via golang-codereviews" <
golang-codereviews@googlegroups.com> wrote:

>
> https://codereview.appspot.com/169970043/diff/1/src/bufio/scan.go
> File src/bufio/scan.go (right):
>
> https://codereview.appspot.com/169970043/diff/1/src/
> bufio/scan.go#newcode133
> src/bufio/scan.go:133: if s.empties > 100 {
> 100 seems low
> and this should certainly be documented
> someone is going to scan a file with >100 blank lines and get really
> confused when it stops scanning partway through
>
> https://codereview.appspot.com/169970043/
>
> --
> You received this message because you are subscribed to the Google Groups
> "golang-codereviews" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-codereviews+unsubscribe@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[r, 2014-11-05 02:56:15]

Hello golang-codereviews@googlegroups.com, rsc@golang.org, adg@google.com,
bradfitz@golang.org (cc: golang-codereviews@googlegroups.com),

Please take another look.

[adg1, 2014-11-05 05:30:39]

LGTM

[r, 2014-11-05 22:57:50]

*** Submitted as https://code.google.com/p/go/source/detail?r=515e1c64bda6 ***

bufio: don't loop generating empty tokens

The new rules for split functions mean that we are exposed
to the common bug of a function that loops forever at EOF.
Pick these off by shutting down the scanner if too many
consecutive empty tokens are delivered.

Fixes issue 9020.

LGTM=rsc, adg
R=golang-codereviews, rsc, adg, bradfitz
CC=golang-codereviews
https://codereview.appspot.com/169970043

[rsc, 2014-11-06 00:35:36]

https://codereview.appspot.com/169970043/diff/40001/src/bufio/scan.go
File src/bufio/scan.go (right):

https://codereview.appspot.com/169970043/diff/40001/src/bufio/scan.go#newcode131
src/bufio/scan.go:131: if len(token) > 0 {
if advance > 0 {

[rsc, 2014-11-06 00:53:50]

https://codereview.appspot.com/170030043 fixes the blank line issue that
Brad pointed out.