New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
encoding/json: no way to preserve the order of map keys #27179
Comments
This was asked in the past. Last week: #27050, and an old issue: #6244, where brad wrote:
While it's unlikely that the current behaviour of the json package will change, a proposal to add some new mechanism that preserves order (similar to the one in the yaml package) may be accepted. It would help, though, to have something more concrete to look at, i.e. a complete, full fledged proposal explaining what the new piece of API would look like. |
My search failed to turn up either of those issues, thanks. I tried to make it clear that this is not about correctness, it's about being nice to human readers. If that's not a relevant concern then we can close this. If there is some chance of a change being accepted then I can make a proposal. |
There is a bit of precedent to making things nicer for human readers. For example, The I imagine there's no way to do this under the hood or by simply adding extra API like Adding a type like |
Similar proposal in the past - keeping the order of the headers in Just like in this case, the big question was where to store the actual order of the map keys. |
Strictly speaking, you can do this today using the raw tokenizer, that |
I think the proposal would look like a flag ( @dsnet Yeah, that solves the input half of the problem, but it's very inconvenient. |
Alternatively, it could output as |
I actually like that a lot. It should probably still be declared in the |
For now it seems like the best answer is a custom type (maybe a slice of pairs) with an UnmarshalJSON method that in turn uses the tokenizer. |
For the people who stumble upon this issue from Google, the following two libraries (pick one) can help you if you need an ordered JSON map: https://gitlab.com/c0b/go-ordered-json Also, for reference, see this common StackOverflow answer: Of course, it would be fantastic if this were eventually part of the standard library, so I'll eagerly await a proposal from someone more proficient than I. |
As per JSON specification, order does not matter. Other technologies, which use JSON, where order matters. It's a much-needed enhancement. |
Feature needed to simplify our implementation on ordered serialization/deserialization. |
We just ran into a case where the order of fields in a JSON file was important, and this code snippet was helpful. However, when order matters not only in the top-level JSON object but also in deeper nested objects, the need to preserve order complicates the code significantly - instead of "just" unmarshaling the object we have to do a series of piece-meal unmarshals to |
Using MapSlice in JSON. type MapItem struct {
Key, Value interface{}
}
type MapSlice []MapItem
func (ms MapSlice) MarshalJSON() ([]byte, error) {
buf := &bytes.Buffer{}
buf.Write([]byte{'{'})
for i, mi := range ms {
b, err := json.Marshal(&mi.Value)
if err != nil {
return nil, err
}
buf.WriteString(fmt.Sprintf("%q:", fmt.Sprintf("%v", mi.Key)))
buf.Write(b)
if i < len(ms)-1 {
buf.Write([]byte{','})
}
}
buf.Write([]byte{'}'})
return buf.Bytes(), nil
} Complete example with unmarshal in Go Playground As a package mapslice-json. |
Hi, I have a question about this issue. I have been looking at the source code a bit and I found that the function sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s }) Here's a bigger snippet: // Extract and sort the keys.
keys := v.MapKeys()
sv := make([]reflectWithString, len(keys))
for i, v := range keys {
sv[i].v = v
if err := sv[i].resolve(); err != nil {
e.error(fmt.Errorf("json: encoding error for type %q: %q", v.Type().String(), err.Error()))
}
}
sort.Slice(sv, func(i, j int) bool { return sv[i].s < sv[j].s }) Why this sort is done in first place? Is there any reason or did I miss something? I was just giving a go and removed line and the Thanks!!! =) |
@dfurtado json encoding should be deterministic. Ranging over a map has an unspecified order, so there isn't a stable order we can use by default. So the encoder falls back to the equivalent of |
A result from manifest v2 contains logs from pipelines. The individual pipelines are stored as an object, thus they have no order. Well, at least in Go, because it doesn't guarantee one when parsing maps, see: golang/go#27179 Unfortunately, this makes the Result.fromV2 method return unpredictable results because the pipeline results are processed in basically a random order. This caused the TestUnmarshalV2Failure test (result_test.go:124) to randomly fail because it expects the ordering to be stable. I decided to fix this by ordering the pipeline results by their name. When this fix is applied, the output from Result.fromV2 is well-defined. Signed-off-by: Ondřej Budai <ondrej@budai.cz>
A result from manifest v2 contains logs from pipelines. The individual pipelines are stored as an object, thus they have no order. Well, at least in Go, because it doesn't guarantee one when parsing maps, see: golang/go#27179 Unfortunately, this makes the Result.fromV2 method return unpredictable results because the pipeline results are processed in basically a random order. This caused the TestUnmarshalV2Failure test (result_test.go:124) to randomly fail because it expects the ordering to be stable. I decided to fix this by ordering the pipeline results by their name. When this fix is applied, the output from Result.fromV2 is well-defined. Signed-off-by: Ondřej Budai <ondrej@budai.cz>
A result from manifest v2 contains logs from pipelines. The individual pipelines are stored as an object, thus they have no order. Well, at least in Go, because it doesn't guarantee one when parsing maps, see: golang/go#27179 Unfortunately, this makes the Result.fromV2 method return unpredictable results because the pipeline results are processed in basically a random order. This caused the TestUnmarshalV2Failure test (result_test.go:124) to randomly fail because it expects the ordering to be stable. I decided to fix this by ordering the pipeline results by their name. When this fix is applied, the output from Result.fromV2 is well-defined. Signed-off-by: Ondřej Budai <ondrej@budai.cz>
A result from manifest v2 contains logs from pipelines. The individual pipelines are stored as an object, thus they have no order. Well, at least in Go, because it doesn't guarantee one when parsing maps, see: golang/go#27179 Unfortunately, this makes the Result.fromV2 method return unpredictable results because the pipeline results are processed in basically a random order. This caused the TestUnmarshalV2Failure test (result_test.go:124) to randomly fail because it expects the ordering to be stable. I decided to fix this by ordering the pipeline results by their name. When this fix is applied, the output from Result.fromV2 is well-defined. Signed-off-by: Ondřej Budai <ondrej@budai.cz>
We ran into this issue, where a third-party API we use requires us to extract an object from a JSON response and return it completely unchanged in a subsequent request body, including not changing the key ordering. Our workaround was to treat the object as an opaque byte slice like:
Not sure if there is any better way to achieve the same thing. Is there any reason why the JSON marshaller cannot do this for []byte types automatically without the need for a custom type and marshal/unmarshal functions? |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
I recently encountered this issue, after investigating the existing solution/project, unfortunately I found that none of them fully met my three needs: preserve order, allow duplicated key, lossless number. So I wrote my own solution, It looks like: data := `{"a": 1.1234543234343238726283746, "b": false, "c": null, "b": "I'm back"}`
result, _ := geko.JSONUnmarshal([]byte(data), geko.UseNumber(true))
output, _ := json.Marshal(result)
fmt.Printf("Output: %s\n", string(output))
// Output: {"a":1.1234543234343238726283746,"b":false,"c":null,"b":"I'm back"}
object := result.(geko.ObjectItems)
fmt.Printf("b: %#v\n", object.Get("b"))
// b: []interface {}{false, "I'm back"} In the case of someone happens to have the same scenario: geko and it's document. |
This comment was marked as duplicate.
This comment was marked as duplicate.
Go team of Google: You should NOT rely on order of JSON Object, it's written in RFC. |
The encoding/json library apparently offers no way to decode data into a interface{} without scrambling the order of the keys in objects. Generally this isn't a correctness issue (per the spec the order isn't important), but it is super annoying to humans who may look at JSON data before and after it's been processed by a go program.
The most popular yaml library solves the problem like this: https://godoc.org/gopkg.in/yaml.v2#MapSlice
The text was updated successfully, but these errors were encountered: