Package scanner
import "go/scanner"
Package scanner implements a scanner for Go source text. Takes a []byte as source which can then be tokenized through repeated calls to the Scan function. Typical use:
var s Scanner
fset := token.NewFileSet() // position information is relative to fset
file := fset.AddFile(filename, fset.Base(), len(src)) // register file
s.Init(file, src, nil /* no error handler */, 0)
for {
pos, tok, lit := s.Scan()
if tok == token.EOF {
break
}
// do something here with pos, tok, and lit
}
Package files
errors.go scanner.goConstants
These constants control the construction of the ErrorList returned by GetErrors.
const (
Raw = iota // leave error list unchanged
Sorted // sort error list by file, line, and column number
NoMultiples // sort error list and leave only the first error per line
)
The mode parameter to the Init function is a set of flags (or 0). They control scanner behavior.
const (
ScanComments = 1 << iota // return comments as COMMENT tokens
AllowIllegalChars // do not report an error for illegal chars
InsertSemis // automatically insert semicolons
)
func PrintError
func PrintError(w io.Writer, err os.Error)
PrintError is a utility function that prints a list of errors to w, one error per line, if the err parameter is an ErrorList. Otherwise it prints the err string.
type Error
Within ErrorVector, an error is represented by an Error node. The position Pos, if valid, points to the beginning of the offending token, and the error condition is described by Msg.
type Error struct {
Pos token.Position
Msg string
}
func (*Error) String
func (e *Error) String() string
type ErrorHandler
An implementation of an ErrorHandler may be provided to the Scanner. If a syntax error is encountered and a handler was installed, Error is called with a position and an error message. The position points to the beginning of the offending token.
type ErrorHandler interface {
Error(pos token.Position, msg string)
}
type ErrorList
An ErrorList is a (possibly sorted) list of Errors.
type ErrorList []*Error
func (ErrorList) Len
func (p ErrorList) Len() int
ErrorList implements the sort Interface.
func (ErrorList) Less
func (p ErrorList) Less(i, j int) bool
func (ErrorList) String
func (p ErrorList) String() string
func (ErrorList) Swap
func (p ErrorList) Swap(i, j int)
type ErrorVector
ErrorVector implements the ErrorHandler interface. It maintains a list of errors which can be retrieved with GetErrorList and GetError. The zero value for an ErrorVector is an empty ErrorVector ready to use.
A common usage pattern is to embed an ErrorVector alongside a scanner in a data structure that uses the scanner. By passing a reference to an ErrorVector to the scanner's Init call, default error handling is obtained.
type ErrorVector struct {
// contains filtered or unexported fields
}
func (*ErrorVector) Error
func (h *ErrorVector) Error(pos token.Position, msg string)
ErrorVector implements the ErrorHandler interface.
func (*ErrorVector) ErrorCount
func (h *ErrorVector) ErrorCount() int
ErrorCount returns the number of errors collected.
func (*ErrorVector) GetError
func (h *ErrorVector) GetError(mode int) os.Error
GetError is like GetErrorList, but it returns an os.Error instead so that a nil result can be assigned to an os.Error variable and remains nil.
func (*ErrorVector) GetErrorList
func (h *ErrorVector) GetErrorList(mode int) ErrorList
GetErrorList returns the list of errors collected by an ErrorVector. The construction of the ErrorList returned is controlled by the mode parameter. If there are no errors, the result is nil.
func (*ErrorVector) Reset
func (h *ErrorVector) Reset()
Reset resets an ErrorVector to no errors.
type Scanner
A Scanner holds the scanner's internal state while processing a given text. It can be allocated as part of another data structure but must be initialized via Init before use.
type Scanner struct {
// public state - ok to modify
ErrorCount int // number of errors encountered
// contains filtered or unexported fields
}
func (*Scanner) Init
func (S *Scanner) Init(file *token.File, src []byte, err ErrorHandler, mode uint)
Init prepares the scanner S to tokenize the text src by setting the scanner at the beginning of src. The scanner uses the file set file for position information and it adds line information for each line. It is ok to re-use the same file when re-scanning the same file as line information which is already present is ignored. Init causes a panic if the file size does not match the src size.
Calls to Scan will use the error handler err if they encounter a syntax error and err is not nil. Also, for each error encountered, the Scanner field ErrorCount is incremented by one. The mode parameter determines how comments, illegal characters, and semicolons are handled.
Note that Init may call err if there is an error in the first character of the file.
func (*Scanner) Scan
func (S *Scanner) Scan() (token.Pos, token.Token, string)
Scan scans the next token and returns the token position, the token, and the literal string corresponding to the token. The source end is indicated by token.EOF.
If the returned token is token.SEMICOLON, the corresponding literal string is ";" if the semicolon was present in the source, and "\n" if the semicolon was inserted because of a newline or at EOF.
For more tolerant parsing, Scan will return a valid token if possible even if a syntax error was encountered. Thus, even if the resulting token sequence contains no illegal tokens, a client may not assume that no error occurred. Instead it must check the scanner's ErrorCount or the number of calls of the error handler, if there was one installed.
Scan adds line information to the file added to the file set with Init. Token positions are relative to that file and thus relative to the file set.