proposal: encoding/json: allow defining which fields will be marshalled #45812

jtorz · 2021-04-28T01:45:28Z

What version of Go are you using (`go version`)?

$ go version
go version go1.16.3 linux/amd64

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (`go env`)?

go env Output

$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/jtorz/.cache/go-build"
GOENV="/home/jtorz/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/jtorz/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/jtorz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build3209792062=/tmp/go-build -gno-record-gcc-switches"

What did you do?

Similar issues: #22480, #11939, #45669

The idea of this proposal is to allow defining which columns will be marshal. It would be useful in many situations where we want to reduce the data size like HTTP responses, or marshalling to cache JSON values.

Currently, with the Standard library when we have something like this, the inner Structures are marshal into empty values.
https://play.golang.org/p/UVjAKNTMqW9

package main

import (
	"encoding/json"
	"fmt"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Jane"}
	var j, _ = json.Marshal(b)
	fmt.Printf("%s\n", j)
}

In the result "a" holds an object with empty values and there are case where thi.

{"a":{"something_1":"","something_2":"","something_3":"","something_4":""},"name":"Joe"}

Solution Proposal

I copied the code from the standard library and did some modifications to allow this. https://github.com/jtorz/jsont

In my module I added the function func MarshalFields(v interface{}, whitelist F) ([]byte, error).

MarshalFields receives a type F map[string]F which works as a whitelist that holds the JSON keys that will be marshal if the key exists in the map, no matter the value, the field is added.

In the next example only the "name" is added to the JSON, all other fields are ignored.

https://play.golang.org/p/ghG6SDyXNLU

In this Example

package main

import (
	"fmt"
	"github.com/jtorz/jsont/v2"
)

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a,omitempty"`
	Name  string `json:"name,omitempty"`
	Hello string `json:"hello,omitempty"`
}

func main() {
	b := B{Name: "Joe", Hello: "This will be ignored"}
	j, _ := jsont.MarshalFields(b, jsont.F{
		"name": nil,
	})
	fmt.Printf("%s\n", j)
}

{"name":"Joe"}

Cases

I considered different scenarios like slices, nested structures, or self-referential structures.

The following cases will use the same structs definitions.

type A struct {
	Something1 string `json:"something_1"`
	Something2 string `json:"something_2"`
	Something3 string `json:"something_3"`
	Something4 string `json:"something_4"`
}

type B struct {
	A     A      `json:"a"`
	Name  string `json:"name"`
	Hello string `json:"hello"`
	Bs    []B    `json:"bs,omitempty"`
}

self-referential Slice (Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Name: "Jane"},
        {Name: "John"},
        {Name: "Janah"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs":   jsont.Recursive,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"name":"Jane"},{"name":"John"},{"name":"Janah"}]}

self-referential Slice (Non Recursive)

b := B{
    Name: "Joe",
    Bs: []B{
        {Hello: "World!"},
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "name": nil,
    "bs": jsont.F{
        "hello": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"name":"Joe","bs":[{"hello":"World!"}]}

Nested Struct (Whole structure)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a":     nil,
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1","something_2":"value2","something_3":"","something_4":""}}

Nested Struct (Specific fields)

b := B{
    Hello: "Janah",
    A: A{
        Something1: "value1",
        Something2: "value2",
    },
}
j, _ := jsont.MarshalFields(b, jsont.F{
    "hello": nil,
    "a": jsont.F{
        "something_1": nil,
    },
})
fmt.Printf("%s\n", j)

OUTPUT:

{"hello":"Janah","a":{"something_1":"value1"}}

The text was updated successfully, but these errors were encountered:

ianlancetaylor · 2021-04-28T05:10:30Z

Can you write in this issue exactly what json.MarshalFields does? It's easier to understand a spec than an implementation. Thanks.

jtorz · 2021-04-29T01:34:42Z

I added more information to the Solution Proposal part

rsc · 2021-05-05T17:51:34Z

/cc @dsnet

dsnet · 2021-05-10T14:59:09Z

For prior art, this seems to have overlap with the field_mask feature from protocol buffers.

seankhliao · 2021-05-10T15:55:42Z

Related #23304

dsnet · 2021-05-10T16:14:53Z

How does this feature interoperate with the fact that json allows custom marshalers and unmarshalers?

jtorz · 2021-05-10T18:15:24Z

I added another interface MarshalerFields, if the type implements that interface it uses that encoder. But there is a precedence of Marshalers, if the same type also implements the Marshaler interface it uses that custom Marshaler as the encoder.

type MarshalerFields interface {
	MarshalJSONFields(whitelist F) ([]byte, error)
}

I didn't consider the decoding in my module, but I think the custom Marshal/Unmarshaler interface should be used and ignore the defined values.

dsnet · 2021-05-10T21:18:17Z

Plumbing down options is a known problem in json and I don't think we should define another interface for another specialized serialization mode.

Even if the problem of plumbing down options was solved, there's the fundamental problem that most implementations will not support some of type of field mask, which makes support for this feature spotty at best.

If the desire is to have semantically correct field mask, then it seems that a better solution is to pre-process JSON input when unmarshaling and post-process JSON output when marshaling to mask out certain fields. It won't be as efficient, but would be guaranteed to be correct. However, if we're willing to accept that this is a pre-process or post-process step, it doesn't seem like this is something that the standard json package needs to support.

networkimprov · 2021-05-16T06:47:01Z

cc @mvdan

gopherbot added this to the Proposal milestone Apr 28, 2021

gopherbot added the Proposal label Apr 28, 2021

ianlancetaylor changed the title ~~proposal: encoding/json allow defining which columns will be marshal~~ proposal: encoding/json: allow defining which columns will be marshal Apr 28, 2021

ianlancetaylor added this to Incoming in Proposals (old) Apr 28, 2021

seankhliao added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 28, 2021

seankhliao removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Apr 29, 2021

acynothia mentioned this issue May 30, 2023

Support configure tag when marshal/unmarshal bytedance/sonic#440

Open

ianlancetaylor changed the title ~~proposal: encoding/json: allow defining which columns will be marshal~~ proposal: encoding/json: allow defining which fields will be marshalled Mar 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

proposal: encoding/json: allow defining which fields will be marshalled #45812

proposal: encoding/json: allow defining which fields will be marshalled #45812

jtorz commented Apr 28, 2021 •

edited

ianlancetaylor commented Apr 28, 2021

jtorz commented Apr 29, 2021

rsc commented May 5, 2021

dsnet commented May 10, 2021

seankhliao commented May 10, 2021

dsnet commented May 10, 2021

jtorz commented May 10, 2021

dsnet commented May 10, 2021

networkimprov commented May 16, 2021

proposal: encoding/json: allow defining which fields will be marshalled #45812

proposal: encoding/json: allow defining which fields will be marshalled #45812

Comments

jtorz commented Apr 28, 2021 • edited

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

Solution Proposal

Cases

self-referential Slice (Recursive)

self-referential Slice (Non Recursive)

Nested Struct (Whole structure)

Nested Struct (Specific fields)

ianlancetaylor commented Apr 28, 2021

jtorz commented Apr 29, 2021

rsc commented May 5, 2021

dsnet commented May 10, 2021

seankhliao commented May 10, 2021

dsnet commented May 10, 2021

jtorz commented May 10, 2021

dsnet commented May 10, 2021

networkimprov commented May 16, 2021

jtorz commented Apr 28, 2021 •

edited

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?