Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

encoding/json: marshaling should have an option to lowercase field names #41448

Closed
bjg2 opened this issue Sep 17, 2020 · 11 comments
Closed

encoding/json: marshaling should have an option to lowercase field names #41448

bjg2 opened this issue Sep 17, 2020 · 11 comments
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@bjg2
Copy link

bjg2 commented Sep 17, 2020

What version of Go are you using (go version)?

$ go version

go version go1.14.6 windows/amd64

Does this issue reproduce with the latest release?

yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

set GO111MODULE=
set GOARCH=amd64
set GOBIN=
set GOCACHE=C:\Users\bjg\AppData\Local\go-build
set GOENV=C:\Users\bjg\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=C:\Users\bjg\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=c:\go
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=c:\go\pkg\tool\windows_amd64
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=D:\resources\init_0\backend\go\common\go.mod
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\bjg\AppData\Local\Temp\go-build390343214=/tmp/go-build -gno-record-gcc-switches

What did you do?

Created a struct (without json annotations) and marshaled it.

What did you expect to see?

I needed field names with lowerCammelCase.

What did you see instead?

I got them with UpperCammelCase.

Extra

To explain a bit more: go JSON decisions create a huge problem in usability that is very simple to solve.

Go forces developers to write capitalized field names if they are to be marshaled into JSON (as they are public fields).

Go never give you the option to choose field names format when marshaling JSON, except via annotations.

Annotations are terrible and are not an option at all. If I am to write JSON annotation for each field that field is called exactly the same but in other format, that's usability hell. Also, when I rename the field, I should rename annotation as well? I should teach all employees that they have to keep the annotations correct? And consider all of this on a huge API project which has hundreds of structs. That's against the very things go is about, which is simplicity and readability of code.

Because of iteroperation with other code, sometimes structs field names need to be formatted into other formats, and that's not up for a discussion. So we have to jump through hoops to get it to work. First, after JSON generation we used a regex to change each JSON key into desired format. Later, we figured out that is not working for maps (where we don't want to change the keys), so we're doing some reflection preprocessing before JSON marshaling. So yeah, now we're doing reflection twice just to marshal JSON...

I am aware that this discussion already existed. I just can't believe that the decision is still to prevent such a basic feature, so I have to reopen it.

@mvdan
Copy link
Member

mvdan commented Sep 17, 2020

huge problem in usability
Annotations are terrible
that's usability hell
can't believe that the decision is still to prevent such a basic feature

Before I reply, I'd suggest that you try to not write issues in frustration. As a maintainer, they are not only harder to read, but also make me less likely to take the time to understand your issue and write a proper reply. We have a code of conduct precisely so that we can all communicate while being respectful to others :)

I should teach all employees that they have to keep the annotations correct?

If you absolutely can't deal with maintaining two names for each field, the answer is usually generating code. For example, Protobuf generates Go code which produces the corresponding field tags for json. If you just need JSON support, you could do that with something much lighter than Protobuf, like https://medium.com/@aniketawati_75058/go-generate-to-automate-json-tag-generation-for-go-structs-2f83bca1f12c.

Code generation is likely a good idea for a large set of people with hundreds of API structs, anyway. That is going to enable you to implement many other features which could otherwise require copying boilerplate and maintaining duplicates.

@mvdan mvdan changed the title json marshaling should have an option to lowercase field names encoding/json: marshaling should have an option to lowercase field names Sep 17, 2020
@mvdan
Copy link
Member

mvdan commented Sep 17, 2020

As for the feature request in particular: adding an option just to lowercase field names is far too narrow. What if someone wants to convert CamelCase to snake_case instead, would that be yet another option? What if someone wants to turn CamelCase into camelcase? What if someone wants to add a prefix to a field name, or strip a prefix? There are just endless possibilities for such transformations, and field tag options aren't nearly powerful enough for this.

@mvdan mvdan added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 17, 2020
@bjg2
Copy link
Author

bjg2 commented Sep 18, 2020

Before I reply, I'd suggest that you try to not write issues in frustration. As a maintainer, they are not only harder to read, but also make me less likely to take the time to understand your issue and write a proper reply. We have a code of conduct precisely so that we can all communicate while being respectful to others :)

That totally makes sense, I agree, I did use frustrated wording and that totally does not help.

If you absolutely can't deal with maintaining two names for each field, the answer is usually generating code. For example, Protobuf generates Go code which produces the corresponding field tags for json. If you just need JSON support, you could do that with something much lighter than Protobuf, like https://medium.com/@aniketawati_75058/go-generate-to-automate-json-tag-generation-for-go-structs-2f83bca1f12c.

Did try it out as well before I wrote this. It still does not solve the problem. For example, if you already have annotation it will not change it - which means if you are changing field name and you want to regenerate the field it will not do. And you still have to generate it eachtime and teach people to do it and expect them to do it, or run it automatically on checkin gate which is complicating the system. It is just a patch.

As for the feature request in particular: adding an option just to lowercase field names is far too narrow. What if someone wants to convert CamelCase to snake_case instead, would that be yet another option? What if someone wants to turn CamelCase into camelcase? What if someone wants to add a prefix to a field name, or strip a prefix? There are just endless possibilities for such transformations, and field tag options aren't nearly powerful enough for this.

I totally agree. My proposal is adding func(name string) string argument which is mapping struct field name into JSON key name. Also, there would be few implementations that go with the json package like json.LowerCammelCase, json.SnakeCase, etc. So in my usecase I would write something like json.Marshal(val, json.LowerCammelCase).

Also, if passing it with each Marshal / MarshalIdent complicates function signature (you need both of these with and without func param), it might be a good idea to be able to create some struct that has a field with this key-mapping value (or nil if no mapping is to be done), and to call Marshal / MarshalIdent on that one with the similar signature as now.

@mvdan
Copy link
Member

mvdan commented Sep 18, 2020

The encoding/json API is stable, like the rest of the standard library, so we absolutely cannot modify existing function signatures.

if you already have annotation it will not change it

I'm not sure I follow. If you have a piece of Go code that generates all of your types, you can make it do whatever you want, including arbitrary field tags. You would not edit or maintain the types directly, instead modifying the generator or its input. This is similar to how Protobuf works.

It is just a patch.

I'm sorry but I strongly disagree. Go is a simple language on purpose. If you want high-level control over your types or code, code generation is the usual answer, and it's also a first-class citizen thanks to go generate. This might all change with generics in the future, but that's not something we can rely on right now.

create some struct that has a field with this key-mapping value

You can't change existing behavior, because that would break existing programs. This includes suddenly treating some fields as special.

Really, the only backwards-compatible option you have in encoding/json would be to add an option to Encoder, similar to #5901 (comment), like Encoder.SetKeyTransform(func(string) string).

Still, beware that the bar for such new options is very high, and encoding/json already has dozens of proposals for new features which we can't accept without exponentially incrementing the complexity of the package and its API. So, given that you can solve this with code generation, I would suggest that you give that a fair shot.

@mvdan
Copy link
Member

mvdan commented Sep 18, 2020

Also, if you do want to suggest API changes to encoding/json, you should turn this into a proposal (https://github.com/golang/proposal). Just beware that, as I said before, the bar for such changes is high.

@bjg2
Copy link
Author

bjg2 commented Sep 18, 2020

I understand all of your comments about stability, that's why I mentioned the signatures. Maybe I haven't elaborated, but the basic idea was to:

  • add a new struct that has such a field to the json package (for now let's call it json.Builder)
  • that struct would have function Marshal(value interface{}) ([]byte, error) with the same signature as now
  • json package would have defaultBuilder without key mapping and json.Marshal would just call defaultBuilder.Marshal. That way everyone that needs special key marshaling would instantiate json.Builder, json.Marshal would reuse the same code as json.Builder.Marshal, and json.Marshal signature would not change, hence noones code would be broken.

About code generation, this one did not work because of mentioned issues https://medium.com/@aniketawati_75058/go-generate-to-automate-json-tag-generation-for-go-structs-2f83bca1f12c. I accept I never built my own code generator and I do understand that it is the only way to have a glimpse of generics before generics.

If I hit some performance issues with double reflection solution (as it works well enough for now) I will try to go down the code generation route, but I think my point stands: I don't think we hit some rare or obscure issue here - I think almost everyone building big APIs have this same issue. I think golang is the best program for web servers, and that many of those web servers communicate with other moving parts in JSON, and have the same issue as we do. I think that standard library should provide solution for such a standard problem.

@sailing0505
Copy link

I have the same concern when I first use the built-in json package.
I'm new to go, and I have spent hours to find out why my custom struct always returns {} after marshal to JSON string. While there is no error happens :(
Until I read the post from the official blog here.
However, I think it might be better if this function(marshal) could raise errors to promote the user in such a situation. because returning {} without error is not helpful to the business code.
Look forward to your comments.

@liuzhenjluccst

This comment was marked as off-topic.

@liuzhenjluccst

This comment was marked as off-topic.

@seankhliao
Copy link
Member

Duplicate of #23027

@seankhliao seankhliao marked this as a duplicate of #23027 Jun 12, 2022
@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Jun 12, 2022
@frobnitzem
Copy link

FWIW, this extremely popular yaml package does exactly what is being requested here by default: https://pkg.go.dev/gopkg.in/yaml.v3. It makes sense not to fight the json and yaml case conventions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

6 participants