You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Problem: It's not currently possible to write a "pure" bufio.SplitFunc function that either (a) returns an empty final token or (b) wants to discard all text beyond some point in the input stream. Instead, a package must implement a bufio.SplitFunc as a method on a state object, as in @robpike's demonstration of how to treat newlines as separators.
Proposed solution: Modify the implementation of Scanner.Scan to check the concrete type of the error returned by the bufio.SplitFunc function. If it's some sentinel value such as io.EOF or, say, a new bufio.EndOfScan, Scanner.Scan should store the returned token, as it does in the normal, non-error case, and return false. A subsequent scan could then continue reading the input stream from where the previous scan left off, making the proposed solution additionally useful for parsing a block of one token type followed by a block of a different token type. The proposed change to Scanner.Scan should not affect any existing code, especially if a new Error type like bufio.EndOfScan is used as the sentinel.
— Scott
The text was updated successfully, but these errors were encountered:
Needs to be thought about carefully but I observe that the test case in scan_test.go for this will loop forever if the explicit counter is removed from the loop.
This is in reference to the golang-nuts thread How to return an empty final token from a bufio.SplitFunc.
Problem: It's not currently possible to write a "pure"
bufio.SplitFunc
function that either (a) returns an empty final token or (b) wants to discard all text beyond some point in the input stream. Instead, a package must implement abufio.SplitFunc
as a method on a state object, as in @robpike's demonstration of how to treat newlines as separators.Proposed solution: Modify the implementation of
Scanner.Scan
to check the concrete type of the error returned by thebufio.SplitFunc
function. If it's some sentinel value such asio.EOF
or, say, a newbufio.EndOfScan
,Scanner.Scan
should store the returned token, as it does in the normal, non-error case, and returnfalse
. A subsequent scan could then continue reading the input stream from where the previous scan left off, making the proposed solution additionally useful for parsing a block of one token type followed by a block of a different token type. The proposed change toScanner.Scan
should not affect any existing code, especially if a newError
type likebufio.EndOfScan
is used as the sentinel.— Scott
The text was updated successfully, but these errors were encountered: