Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: encoding/json: "nonil" struct tag to marshal nil slices and maps as non-null #27589

Open
pjebs opened this issue Sep 10, 2018 · 47 comments · May be fixed by #27813
Open

proposal: encoding/json: "nonil" struct tag to marshal nil slices and maps as non-null #27589

pjebs opened this issue Sep 10, 2018 · 47 comments · May be fixed by #27813

Comments

@pjebs
Copy link
Contributor

pjebs commented Sep 10, 2018

There have been many reports about encoding a nil slice or map as [ ] or { } respectively.

See: #2278

Currently it is encoded as null. This is a source of bugs in many non-theoretical cases (eg consumers of JSON APIs). There are many issues on Google about others falling into this pitfall so it is at least not uncommon.

It would be nice if a struct tag can be used to signify that we intend the nil slice or map to be treated as an empty slice or map respectively for the purposes of marshalling.

Current solutions include: https://github.com/helloeave/json. This is not a nice solution because it requires the assistance of a third-party package.

This is how it's implemented in third party lib: homelight/json@7b6e442

My PR can be found here: #27813 (subject to an agreed struct tag name, which when decided, can be updated in the PR)

@mvdan mvdan changed the title Feature request: Struct tag for json to marshall null slices/maps as non-nil proposal: encoding/json: struct tag to marshal nil slices and maps as non-null Sep 10, 2018
@gopherbot gopherbot added this to the Proposal milestone Sep 10, 2018
@mvdan
Copy link
Member

mvdan commented Sep 10, 2018

I've retitled this to be a proposal.

I'm not sure if it's a good idea, but it's certainly feasible and wouldn't add much new API. /cc @bradfitz @dsnet

@pjebs
Copy link
Contributor Author

pjebs commented Sep 12, 2018

For slices and maps, it should be noted that the proposed struct tag is mutually exclusive to omitempty. For backwards compatibility, if both omitempty and the new struct tag is present, then omitempty takes precedence.

@pjebs pjebs closed this as completed Sep 13, 2018
@pjebs pjebs reopened this Sep 13, 2018
@pjebs
Copy link
Contributor Author

pjebs commented Sep 22, 2018

Here is my pull request. I hope it gets accepted because I believe it adds value to the "Go" programming language.

#27813

@pjebs
Copy link
Contributor Author

pjebs commented Sep 22, 2018

I called the struct tag: nonil

@bradfitz
Copy link
Contributor

"heed" is a unique word, at least in terms of Go's existing API.

If this is accepted (once it's reviewed together with all the existing JSON proposals), I suspect we'd want to change "heed" to something else.

@pjebs
Copy link
Contributor Author

pjebs commented Sep 24, 2018

I scoured the dictionary and thesaurus for appropriate words that were short and exemplified the intention but couldn't find anything. Hopefully a better word can be found because I was initially hesitant of the word too - although admittedly it is growing on me.

@pjebs
Copy link
Contributor Author

pjebs commented Sep 24, 2018

@bradfitz Any idea of the timeframe to when all the JSON proposals will be examined? Some of them are now 2+ years. The pivotal proposal is 7+ years.

@QtRoS
Copy link
Contributor

QtRoS commented Sep 25, 2018

@bradfitz @pjebs what about nullsafe or nullfree or nonull?
+1 that heednull is unacceptable.

@pjebs
Copy link
Contributor Author

pjebs commented Sep 26, 2018

For Go2, omitempty should become omitzero. The zero value and what is colloquially called "empty" are different for maps and slices.

@rsc
Copy link
Contributor

rsc commented Oct 3, 2018

Will add this to my JSON scan for this cycle.

@rsc
Copy link
Contributor

rsc commented Oct 3, 2018

Maybe emptynil.

@pjebs
Copy link
Contributor Author

pjebs commented Oct 3, 2018

HEED
verb
1.
pay attention to; take notice of.

noun
1.
careful attention.

@pjebs
Copy link
Contributor Author

pjebs commented Oct 3, 2018

Seems like precisely the right word. It's not part of the Go vocabulary, but it seems like a great addition in various future contexts for programming languages in general.

eg. heed memory, heed security etc

@rsc
Copy link
Contributor

rsc commented Oct 10, 2018

heednull means "pay attention to null" by not respecting that the value is null and instead printing something other than "null"? That seems backward to me.

@pjebs
Copy link
Contributor Author

pjebs commented Oct 10, 2018

If it was called heednil instead, then one can argue that the slice or map was nil, and you are instructing the encoder to "pay attention to it".

I then went from that step to heednull because it flowed off the tongue better. The null in heednull was not really a reference to the eventual json output null.

Perhaps due to the confusion, and ambiguity, heednil is closer to being "Go-like".

@rsc
Copy link
Contributor

rsc commented Oct 17, 2018

On hold for JSON sweep.

@pjebs
Copy link
Contributor Author

pjebs commented Oct 17, 2018

What does that mean?

@dsnet
Copy link
Member

dsnet commented Oct 17, 2018

The json package already has a bloated API surface. Any additions need to carefully considered together with all the other proposals. I alluded to the need for this: #27813 (comment)

The "JSON sweep" is a sweep through all the JSON proposal trying to come up with a comprehensive yay/nay to each proposal in an effort to balance features desired with not adding too much technical debt.

@tommie
Copy link

tommie commented Dec 31, 2018

Current solutions include: https://github.com/helloeave/json

This fork (which sadly I can't use because AppEngine doesn't have sync.Map, it seems) sets an encoder-wide option instead of using a struct tag. It feels to me that either an entire system is fine with having null, or there is an expectation that all slices are non-null. Why is the proposal to do this as a struct tag instead of an encoder option?

@QtRoS
Copy link
Contributor

QtRoS commented Dec 31, 2018

@tommie in my opinion the problem is that any system doesn't work in isolation. Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

@tommie
Copy link

tommie commented Dec 31, 2018

Complex software has complex integrations with different technologies and different expectations. Global flag is not versatile enough.

That is a non-answer, IMHO (and somewhat belittling). I can see different complex software requiring different options, and AFAIK, this thread has not discussed the already existing solution yet.

  • Say you have a frontend that needs non-null results, and a backend that wants null results. It's not per-field, but per encoder invocation.
  • Say most fields are "arrays", but one field is an optional array. Then it's per-field.

Defining what the default behavior is could still be an encoder option, while allowing per-field overrides. The two solutions are not mutually exclusive.

The problem I'm trying to solve is interoperability with Javascript. Passing null means my browser-side code needs to handle an empty array in a non-idiomatic way, leading to more fragile code. It seems to me @pjebs is trying to solve a different problem with the struct tag solution, and I'm curious what that is.

@gopherbot
Copy link

Change https://golang.org/cl/136761 mentions this issue: encoding/json: added "nonil" option for json encoding (slice/maps)

@as
Copy link
Contributor

as commented May 16, 2020

I provide an explanation in the similar issue here #37711 why I think a tag is not the right way to address this problem even in a third-party package.

@leaxoy
Copy link

leaxoy commented Oct 27, 2020

Any progress on this, this is a great proposal.

@mvdan
Copy link
Member

mvdan commented Oct 27, 2020

@leaxoy see #27589 (comment).

We'll update the many json proposals when we have news. Don't hold your breath because it's a big project, but it's in progress. Until then, asking for progress or an ETA won't really help.

@benitogf
Copy link

benitogf commented Nov 9, 2022

is there any other approach currently for this one?

john-floren-gravwell pushed a commit to john-floren-gravwell/gravwell that referenced this issue Apr 21, 2023
Javascript really does not like to get a null when it expects an empty array,
but that's what Go will send if your struct contains e.g. a []string and
you don't explicitly initialize the slice. These custom marshallers make sure
we send an empty array, making the frontend's life easier.

Go desperately needs some way to tell the JSON encoder that we want this
behavior, see:

golang/go#37711
golang/go#27589
golang/go#27813
@VojtechVitek
Copy link

I just wrote this little package, which recursively initializes all nil slices in a given object using reflect pkg. This effectively forces json.Marshal() to render empty array [] instead of null values.

https://github.com/golang-cz/nilslice

Might be useful for some, until we have a proper solution merged to encoding/json stdlib pkg.

import "github.com/golang-cz/nilslice"
type Payload struct {
	Items []Item `json:"items"`
}

payload := &Payload{}

b, _ = json.Marshal(nilslice.Initialize(payload))
fmt.Println(string(b))
// {"items": []}

VojtechVitek added a commit to webrpc/gen-golang that referenced this issue Aug 23, 2023
… in JSON

Golang issue: golang/go#27589

This option will enforce server to encode nil slices as empty arrays in JSON.

This is helpful mainly in JavaScript and TypeScript clients, where a response
field of type array should never be assigned to `null` value..

Example:

func (s *Server) GetItems(ctx context.Context) ([]*Items, error) {
	var items []*Item

	// Will send {"items": []} in JSON response instead of {"items": null}.
	return items, nil
}
VojtechVitek added a commit to webrpc/gen-golang that referenced this issue Aug 23, 2023
… in JSON

Golang issue: golang/go#27589

This option will enforce server to encode nil slices as empty arrays in JSON.

This is helpful mainly in JavaScript and TypeScript clients, where a response
field of type array should never be assigned to `null` value..

Example:

func (s *Server) GetItems(ctx context.Context) ([]*Items, error) {
	var items []*Item

	// Will send {"items": []} in JSON response instead of {"items": null}.
	return items, nil
}
VojtechVitek added a commit to webrpc/gen-golang that referenced this issue Aug 23, 2023
… in JSON (#32)

Golang issue: golang/go#27589

This option will enforce server to encode nil slices as empty arrays in JSON.

This is helpful mainly in JavaScript and TypeScript clients, where a response
field of type array should never be assigned to `null` value..

Example:

func (s *Server) GetItems(ctx context.Context) ([]*Items, error) {
	var items []*Item

	// Will send {"items": []} in JSON response instead of {"items": null}.
	return items, nil
}
jmendesky added a commit to sky-uk/kfp-operator that referenced this issue Sep 4, 2023
Until go JSON marshalling supports golang/go#27589, initialising a slice
is the only way to marshal empty arrays into JSON instead of null values.
@dsnet
Copy link
Member

dsnet commented Oct 6, 2023

Hi all, we kicked off a discussion for a possible "encoding/json/v2" package that addresses the spirit of this proposal.
By default v2 proposes that a nil slice or map would marshal as an empty JSON array or object.

For flexibility, it provides the ability to alter this behavior with:

  • json.Marshal(v, json.FormatNilSliceAsNull(false)) at the marshal-call level
  • struct{ SliceField []T `json:",format:emitnull"` } at the struct-field level

@pkierski
Copy link

pkierski commented Mar 7, 2024

I just wrote this little package, which recursively initializes all nil slices in a given object using reflect pkg. This effectively forces json.Marshal() to render empty array [] instead of null values.

https://github.com/golang-cz/nilslice

Based on this idea I've prepared more complex solution: The same applies for nil maps (initialized as empty map, rendered as {} instead of null). And also does it recursively for any non-cyclic structures (for cyclic structured crashes as json.Marshal do). https://github.com/pkierski/niltoempty

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging a pull request may close this issue.