Comment 1:

In particular, I used this in my jsonpointer implementation as it allowed me to get the
information out *significantly* faster and with less memory overhead (i.e. didn't have
to construct a big map[string]interface{} of all the parts and walk it).
This is what I get out of it, in practice:
Using only what's in encoding/json (blob of JSON to map):
BenchmarkLargeMap         20     103041502 ns/op      18.83 MB/s     8139008 B/op     220394 allocs/op
Using the scanner approach so I can throw away all the parts that aren't interesting and
stop when I get to the part I'm seeking.
BenchmarkLarge3Key    100000         24502 ns/op    79194.52 MB/s       7258 B/op         89
allocs/op
BenchmarkLargeShallow       2000        876675 ns/op    2213.44 MB/s       33155 B/op       2718
allocs/op
BenchmarkLargeMissing         50      35058577 ns/op      55.35 MB/s      922645 B/op     101772
allocs/op
BenchmarkLargeIdentity   5000000           711 ns/op    2726464.30 MB/s      566 B/op          5
allocs/op
BenchmarkLargeBest    500000          3893 ns/op    498379.24 MB/s      1445 B/op         24
allocs/op
I've got another project that is needing the incremental scan right now which is why
this just came up.
In the meantime, I'm occasionally pulling in go's json improvements and exposing the
scanner.

Comment 2:

I am skeptical that we want to commit to implementation details like the Scanner.
If you are decoding into a struct it will already throw away the parts that aren't
interesting.

Labels changed: added priority-later, removed priority-triage.

Status changed to Thinking.

Comment 3:

I can understand why exposing, documenting, etc... this stuff could be a longer term
burden.
My particular uses don't involve structs, though.  The data processing is more dynamic
and keys being accessed are user-supplied.  I used the scanner as part of evaluating an
expression to dig data out of, in some cases, large JSON objects.  The first version
(Map above) was necessary using only encoding/json where I have to parse the entire
thing into a map[string]interface{} and then traverse that structure to find the key I
wanted.  The rest are how I do it with scanner.
I also needed to do some streaming transforms of JSON for which I *almost* used the
scanner I'd exposed, but since I was only doing simple substitution in strings, I used a
more simple state machine.

Comment 4:

Labels changed: added go1.3maybe.

Comment 5:

Labels changed: added release-none, removed go1.3maybe.

Comment 6:

Labels changed: added repo-main.

Comment 7:

This is related to issue #6050 ("encoding/json: provide tokenizer").

CL https://golang.org/cl/11651 mentions this issue.

CL https://golang.org/cl/9073 mentions this issue.