proposal: encoding/json: add omitzero option #45669
Comments
seankhliao
changed the title
seankhliao
changed the title
|
Hold for #22480. |
|
Placed on hold. |
|
This proposal has been added to the active column of the proposals project |
|
Personally I don't like baking the omission policy into struct tags. Things like the |
Say we go down that path and I write a type like this: type Null[T any] struct {
Valid bool
Value T
}
func (n Null[T]) MarshalJSON() ([]byte, error) {
if !n.Valid {
return nil, nil
}
return json.Marshal(n.Valid)
}Now this type can only be used as an |
|
@icholy I believe the return-nil option would be instead of |
|
@joeshaw having a |
|
@icholy Returning However, @dsnet points out in #52803 (comment) that |
|
@joeshaw I want to have a general purpose |
|
@icholy So you return string |
|
@mitar my |
|
True. I think this is a broader issue: |
Bugs in 2 and 3 can be fixed as follows: // fast path omitEmpty
if f.omitEmpty && isEmptyValue(fv) {
continue
}
omitEmptyResetLocation := e.Len()
e.WriteByte(next)
if opts.escapeHTML {
e.WriteString(f.nameEscHTML)
} else {
e.WriteString(f.nameNonEsc)
}
opts.quoted = f.quoted
startLen := e.Len()
f.encoder(e, fv, opts)
newLen := e.Len()
if f.omitEmpty && (newLen-startLen) <= 5 {
// using `Next` and `bytes.NewBuffer` we can modify the end of the
// underlying slice efficiently (without doing any copying)
fullBuf := e.Next(newLen) // extract underlying slice from buffer
switch string(fullBuf[startLen:newLen]) {
case "false", "0", "\"\"", "null", "[]":
// reconstruct buffer without zero value
e.Buffer = *bytes.NewBuffer(fullBuf[:omitEmptyResetLocation])
continue
default:
// reconstruct original buffer
e.Buffer = *bytes.NewBuffer(fullBuf)
}
}
next = ','Feature 4 can be added as follows: // Emptyable is the interface implemented by types that
// can provide a function to determine if they are empty.
type Emptyable interface {
IsEmpty() bool
}
var emptyableType = reflect.TypeOf((*Emptyable)(nil)).Elem()
func isEmptyValue(v reflect.Value) bool {
// first check via the type if `Emptyable` is implemented
// this way, we can prevent the more expensive `Interface`
// conversion if not required
if v.Type().Implements(emptyableType) {
if v, ok := v.Interface().(Emptyable); ok {
return v.IsEmpty()
}
}
switch v.Kind() {
case reflect.Array, reflect.Map, reflect.Slice, reflect.String:
return v.Len() == 0
case reflect.Bool:
return !v.Bool()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return v.Int() == 0
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return v.Uint() == 0
case reflect.Float32, reflect.Float64:
return v.Float() == 0
case reflect.Interface, reflect.Ptr:
return v.IsNil()
}
return false
}NOTE: Instead of adding a NOTE2 (edit): by replacing @joeshaw I tried to make this analysis as complete as possible. Please let me know in case I missed something. |
|
/cc @dsnet for any thoughts about how we might want to proceed here. |
|
@rcs @joeshaw @dsnet #53425 is an implementation that solves 'bug' 3 (as described above). |
|
@inteon, the difficult lies not in the implementation but
Let's step back and ignore how the In a brand new world, how would we like object members to be omitted?
Which is correct? Omit based on Go valueLet's suppose there was a Properties of this approach:
Omit based on JSON valueLet's suppose there was a Properties of this approach:
ComposabilityBoth In many cases, the behavior of CompatabilityThe existing semantics of We would need to do one of the following:
Personally, I like the names of |
|
@dsnet Thank you for your extended answer Ok, so 'fixing' I generally agree with your assessment, but I think that it is important to note that even when using a new major version of the Am I correct to assume that all work on a possible json v2 is done here: https://github.com/go-json-experiment/json (looks very promising)? Could you give a timeline on when you would be open to receive feedback/ PRs on this future work? |
|
I'd prefer a omitzero with omitgozero semantics. Then an omitnull which omits any value which would have marshalled to null. |
Agreed. This is something I'm concerned about. It's unclear whether this is an advantage or disadvantage, but if the new "json" package redefines the meaning of
This is just an experimental project worked on by @dsnet, @mvdan, @johanbrandhorst, @rogpeppe, and @ChrisHines. We are not operating in any official capacity (other than @rsc being aware of the work). This work will be presented in the future as a Go proposal when it's ready. Even then, the possible results are highly varied. The project started off somewhat ad-hoc when @mvdan tweeted his thoughts about a hypothetical v2 "json" package. From the beginning, the project had an ambitious scope trying to address most of the open issues with the JSON package in a unified way such that the features are mostly orthogonal and don't interact poorly with each other. Progress has been slow since none of us work on Go full-time, and contribute on the side. For myself, @tailscale has been very generous in giving me the freedom to work on this aside from my other duties there. Of relevance to this issue, |
|
Met with @dsnet and others yesterday. Let's put this on hold until we figure out what to do with their work. |
|
Placed on hold. |
|
As someone currently vendoring I'd also like to echo @smikulcik and say that while new tags like For example, only allowing omission based on predetermined reasons will force a dev to essentially lie in type EphemeralTime time.Time
type SomeType struct {
// ValidUntil is only present while the thing is valid
ValidUntil EphemeralTime `json:"valid_until,omitzero"`
}
func (t EphemeralTime) IsZero() {
if t.Now().After(t) {
return true // Lie just to get it omitted
}
// Actual zero check
}While that's contrived for demonstration, there are other use-cases like non-zero defaults you may want to exclude. If, instead, there was some mechanism for |
|
I agree with @marwatk . It is not an uncommon case. Think of serializing an application configuration file. That configuration is likely full of non-zero default values. But when serializing that configuration i would not want to include everything, only the customized fields. It is much easier to maintain that way knowing that only the things i customize are listed. The use case here would be a configuration front end that reads in current configuration, then merges in your changes (from say a web page) then serializes that out to the configuration file. |
|
Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal. See the "omitzero" struct tag option under the "Struct tag options" section, which omits a struct field if it is the zero Go value (or has a |
|
I don't get it. Couldn't we just support returning NIL in a custom MarshalJSON method? Returning NIL results in an error anyway at the moment. I wish you could make that example work: https://go.dev/play/p/Cx7e0T_D30v |
The
omitemptyjson tag is kind of confusing to use when working with nested structs. The following example illustrates the most basic case using an empty struct for argument's sake.The "EmptyStruct" field is a struct without any fields and can be empty, that is equal to it's zero value. But when I try to marshal it to json the field is still included in the resulting json object. Reading the encoding/json documentation about the definition of empty it does not mention empty structs:
It feels weird that adding the
omitemptytag to struct fields is allowed if it will never have the desired effect. If structs will never be considered empty shouldn't there at least be a compiler warning when adding this tag to a struct field?Working with json time.Time
This behavior causes some confusion when working with
time.Timein json structures. Go's zero values for primitive types are fairly reasonable and "guessable" from a non-gopher point of view. But the zero value fortime.Time, January 1, year 1, 00:00:00.000000000 UTC, is less common. A more(?) common zero value for time is January 01, 1970 00:00:00 UTC. To avoid confusion when working with json outside the world of go it would be nice to have a way to omit zero value dates.A commonly suggested workaround to this problem is to use pointers, but pointers might be undesirable for a number of reasons. They are for example cumbersome to assign values to:
The
time.Timedocumentation also recommends to pass the type as value, not pointer since some methods are not concurrency-safe:This playground example illustrates three different uses of empty structs where I'd expect the fields to be excluded from the resulting json. Note that the
time.Timetype has no exposed fields and uses thetime.Time.NarshalJSON()function to marshal itself into json.Solution
Would it make sense to add a new tag, like
omitzero, that also excludes structs?There's a similar proposal in #11939, but that changes the definition of empty in the
omitemptydocumentation.The text was updated successfully, but these errors were encountered: