encoding/gob: permit registering duplicate names #36345

dsnet · 2020-01-01T00:41:05Z

In go1.9, type aliases were introduced. The purpose of type aliases is to permit the movement of types from one location to another. This implies:

that the real name of the type can change over time
that there can be multiple names for the exact same Go type

The use of type aliases interacts poorly with gob-encoded data stored on disk that represent interface values as gob was a developed in an era where there was one-to-one relationship between types and names.

Consider the following examples.

At some point in the past, we gob-encoded is stored on disk using something like:

type Rect struct { ... }
func (r Rect) Area() int { ... }
type Any struct {
	Shape interface{ Area() int }
}

func main() {
	gob.Register(Rect{})

	var b bytes.Buffer
	e := gob.NewEncoder(&b)
	e.Encode(Shape{Rect{2, 5}})
	
	// assume that b is stored on disk
	fmt.Printf("%q\n", b.String())
}

On the decoder side, this works fine with code like:

type Rect struct { ... }
func (r Rect) Area() int { ... }
type Any struct {
	Shape interface{ Area() int }
}

func main() {
	// Assume that the contents of r is loaded from disk as created by:
	// https://play.golang.org/p/dt0pW47u17P
	r := strings.NewReader(...)

	gob.Register(Rect{})

	d := gob.NewDecoder(r)
	v := new(Shape)
	fmt.Println(d.Decode(v)) // got nil; works as expected
}

At some point in the future, suppose we move Rect, now decoding fails:

type Rect = Rect2 // this is now a type alias to a different name
type Rect2 struct { ... }
func (r Rect2) Area() int { ... }
type Any struct {
	Shape interface{ Area() int }
}

func main() {
	// Assume that the contents of r is loaded from disk as created by:
	// https://play.golang.org/p/dt0pW47u17P
	r := strings.NewReader(...)

	gob.Register(Rect{})

	d := gob.NewDecoder(r)
	v := new(Shape)
	fmt.Println(d.Decode(v)) // got `gob: name not registered for interface: "main.Rect"`, expected nil
}

The problem is that the full type name is encoded in the gob wire data. Trying to decode the data at a later date does not know about the movement of the type due to type aliases.

Given that problem, one would think that you could just use gob.RegisterName to manually register the old name:

gob.RegisterName(Rect{})
gob.RegisterName("main.Rect", Rect{})

However, that panics with:

panic: gob: registering duplicate names for main.Rect2: "main.Rect2" != "main.Rect"

I propose the following changes:

That gob be better documented about the interaction of type aliases and serialization of interface values.
That gob.RegisterName be relaxed to permit duplicate names. However, this can cause confusion in the implementation as to which name to use when marshaling. Alternatively, we could add a gob.RegisterAlias that makes it clear that the duplicate names are aliases.

The text was updated successfully, but these errors were encountered:

dmitshur · 2020-01-03T19:08:27Z

/cc @robpike per owners.

ionous · 2020-12-10T01:30:47Z

@dsnet - at the end of your issue, you said:

Given that problem, one would think that you could just use gob.RegisterName to manually register the old name:
gob.RegisterName(Rect{})
gob.RegisterName("main.Rect", Rect{})

That first line doesnt work because RegisterName takes two parameters. Did you mean this?

gob.Register(Rect{})
gob.RegisterName("main.Rect", Rect{})

because that does raise:

panic: gob: registering duplicate names for main.Rect2: "main.Rect2" != "main.Rect"

When types are moved with aliases the only way to inform gob is to leave out the first line:

// gob.Register(Rect{})  --- ng. Rect is now an alias to Rect2{}
gob.RegisterName("main.Rect", Rect{})

or, this works the same and might be clearer:

gob.RegisterName("main.Rect", Rect2{})  // Rect has moved to Rect2

That doesn't seem too terrible because the code also had to be altered to add the alias. But, It does feel a little unexpected ( re: better docs ), and it does "lock" the type forever to the earlier name.

I just encountered similar issues trying to implement transparent type migration... "upgrading" to new types while reading/writing.

To support type migration, it might be nice ( like your RegisterAlias()? ) if the decoder allowed multiple names to map to the same type. That would allow "main.Rect" and "main.Rect2" names to be loaded into a "Rect2" struct. If the encoder still always used a type's actual name ( ie. "main.Rect2" ) that would effectively move the type after a read and write.

I wasn't able to find similar requests, but encapsulating the registry and creating the de/encoder(s) from it could also do the trick.

// possible syntax:
var oldFormat, newFormat gob.Registry 
oldFormat.RegisterName("main.Rect", Rect2{})
newFormat.Register(Rect2{})
dec, enc:= oldFormat.NewDecoder(), newFormat.NewEncoder()

As a side benefit, registry memory could be released once it's no longer needed. And... taking this a step further... if that registry object exposed a way to translate type names into gob.Decoder and gob.Encoder interfaces... that would allow a level of flexibility for stream translation and migration that's not currently available.

deepch · 2022-07-25T13:43:50Z

please add RegisterAlias

dmitshur added FeatureRequest NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Jan 3, 2020

dmitshur added this to the Backlog milestone Jan 3, 2020

dsnet mentioned this issue Nov 19, 2021

proposal: spec: add sum types / discriminated unions #19412

Open

dsnet mentioned this issue Feb 18, 2022

types/logger, logtail: add mechanism to do structured JSON logs tailscale/tailscale#3969

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

encoding/gob: permit registering duplicate names #36345

encoding/gob: permit registering duplicate names #36345

dsnet commented Jan 1, 2020 •

edited

dmitshur commented Jan 3, 2020

ionous commented Dec 10, 2020

deepch commented Jul 25, 2022

encoding/gob: permit registering duplicate names #36345

encoding/gob: permit registering duplicate names #36345

Comments

dsnet commented Jan 1, 2020 • edited

dmitshur commented Jan 3, 2020

ionous commented Dec 10, 2020

deepch commented Jul 25, 2022

dsnet commented Jan 1, 2020 •

edited