I believe your understanding of the API is incorrect.

Next scans the next token and returns its type.

Token returns the next token. So the code is doing what is expected.

Here is something which is more on the lines of what you are expecting -

func main() {
  reader := strings.NewReader("<div class=\"hello\">SomeText</div>")
  tokenizer := html.NewTokenizer(reader)
  tokenizer.Next()
  fmt.Printf("%s\n", tokenizer.Raw())
  fmt.Printf("%s\n", tokenizer.Raw())
}

But Raw return slice of bytes, while Token returns an instance of Token struct.

It's ok that Next changes the internal state of the object, but I don't feel that it's kind of logical that Token changes the state.

Also, I think that documentation is a little bit incorrect.

Next scans the next token and returns its type.

That's correct

Token returns the next token.

However, Token doesn't return the next token. It wraps the token you're currently on into struct and returns it to you. It doesn't somehow advance the iterator.

Basically, why I think that current logic is incorrect? Because since Token just wraps current 'state' into struct, it's weird for me that after wrap, internal state is messed.

@namusyaka

@agnivade Thanks for pinging. I would like to work on this issue after resolving some crashing issues.