It seems to me this will only be useful if you litter your data structure with fields of type json.Node. That seems inconvenient. Useful for error checks but not the way you want your data to be represented in the long term.

Another possibility would be to define a new method, along the lines of UnmarshalJSON, that takes an offset as an argument. That lets the user of the package choose how to handle the offset.

Either way it's not clear to me this is worth the additional complexity in the standard library package.

It seems to me this will only be useful if you litter your data structure with fields of type json.Node.

My intention was to have this an interface analogous to unmarshalling to an interface{}. The way I've been using it is to unmarshal twice, once into a struct with json tags and once into a json.Node, effectively producing two trees. If you walk down both trees in sync you can simultaneously access both the unmarshalled data for validation and the offsets from the Node tree. A nice side effect of this is you can access the original string that was decoded and use it to generate warnings (e.g. if you have a field representing file permissions and the user put a value in the wrong base like 0644 instead of 420, the decimal equivalent). Furthermore, because all the json gets decoded into the Node, you can check for things like unused fields.

Another possibility would be to have the Value field of the Node structure be the struct you want to unmarshal into, and also add a Children field (being either nil for leaves, map[string]Node for objects or []Node for arrays), but I wanted to get feedback on the simpler case first before implementing it. It's also unclear what the interface for this would look like? A separate UnmarshalWithOffsets function? Unmarshalling to a Node with a struct already set for Value?

Another possibility would be to define a new method, along the lines of UnmarshalJSON, that takes an offset as an argument. That lets the user of the package choose how to handle the offset.

This would require every type to implement the interface, which would lead to a huge amount of boilerplate code. I'm also not sure I follow how this would be used? I'm imagining something like:

type foo Foo
func (f *Foo) UnmarshalJSONOffset(data []byte, offset int) error {
    // record offset somewhere for later use
    foo f2 := foo(*f)
    if err := json.Unmarshal(data, f2); err != nil {
        return err
    }
    *f = Foo(f2)
}

But the problem is that when json.Unmarshal gets called the offsets it uses will be relative to the start of the data passed to UnmarshalJSONOffset, not the start of the entire json blob. I suppose you could add something like json.UnmarsalWithBaseOffset which could automatically add the offset for you, but that seems somewhat messy.

Over in #16426 minux suggested a Validator interface that could accept offset information, which might also help here.

I don't understand your suggestion of unmarshaling twice. If you do that at top level you will just get the top level offset. If you do that at every level then you can keep track of the offset yourself. How would your suggestion work?

If you do that at top level you will just get the top level offset.

When unmarshalling to a Node, its children (if any) are also Nodes (either via a map[string]Node or []Node). Thus every node in the tree is a Node and has offsets; you don't get just the top level offset.

I think I understand your idea better now. It would only save relative offsets for each element and leave computing absolute offsets to the user. Correct? I'm still of the opinion this would need a lot of boilerplate code and would force the user to either embed the offset information in their types or use a global to record the offsets.

Over in #16426 minux suggested a Validator interface that could accept offset information, which might also help here.

I like the Validator idea, but because it needs to validated while being unmarshalled, it cannot check for conflicts if two sections of the json depend on each other (i.e. one references the other). Because json.Unmarshal only returns one error[], it would be impossible to allow multiple validation errors to be returned. This also means you can't return warnings, since json parsing could stop after hitting the first error. Finally, it doesn't allow for the possibility of detecting extra keys. Providing an interface to get offset information from unmarshalling gives users more flexibility.

Offset into strings for JSON is good debug or recovery information potentially.

{ "keyA": "blah", "keyB": "blah", (<- repeated many times) keyABCDEFGH": "blah", "keyABCEFGI": "blah", <- so on }

A bug: you get this instance of JSON once every two weeks with that missing quote. This would be a nightmare to track down with bad or no logging assuming a large dataset - getting the exact malformed JSON point makes knowing "hey a client sent this bad huge data set, they have a bug" provable.

The proposal is a superset of this though, an arbitrary matching of string character index to JSON data symbols for processing beyond just signaling error handling.

"If you look at the original JSON string, each map and slice in the unmarshaled struct and each of their elements and their values point to each of these character indexes" I think is what is being asked for.

A solution could be a method that holds onto the work by returning an independent map:

type JSONSymbol string
type StringOffset int64
func UnmarshalWithOffset(data []byte, v interface{}) (map[JSONSymbol]StringOffset, error)

Then you iterate through the map or directly lookup the JSON symbols you care about. The documentation would have to define the JSONSymbol encoding. Ordering could be part of the JSONSymbol encoding.

This is a specialized use of JSON. I think it would not be too difficult to write a third party implementation to do the work a second time (which leaves the standard library as is):

func OffsetsForUnmarshal(jsondata string, valid YourUnmarshalType) ([]string, map[string]int64)

Determining last character for an object is looking up next object first character, minus one.

I experimented with a map solution as well, except it mapped from the addresses of the unmarshalled data to their offsets instead of mapping from strings to offsets. This avoids having to specify a special encoding for the JSONSymbol. I think this proposal is cleaner and more similar to the existing api, which is why I chose to propose it instead.

This seems largely about validation, which had a proposal that was rejected (#16426), so it seems there is little need for this. Also, as you say, errors already do deliver offset information.

While I don't disagree that validation might be out of scope for the core library, I do think that this mechanism (or something similar) is necessary. To take the example of validation - to create that external validation package would require either support from the core JSON package or an alternate implementation of the unmarshaller. It seems largely a waste of time (and almost certainly error-prone) to rewrite the JSON unmarshaller just for the purpose of exposing more information to the consumer. Modifying the core library to expose this information, on the other hand, is a relatively small change and enables external packages to greatly extend the functionality.