My first attempt was simply to re-use a single Decoder instance and feed in multiple streams. This doesn't work, and results in a "extra data in buffer" (or "duplicate type received" with b3235b7) error. The Decoder does not expect to receive type definitions multiple times.

One idea would be to re-use a Decoder instance anyway, and ignore the duplicate type definitions. This is simple enough:

--- a/src/encoding/gob/decoder.go
+++ b/src/encoding/gob/decoder.go
@@ -57,7 +57,7 @@ func NewDecoder(r io.Reader) *Decoder {
 // recvType loads the definition of a type.
 func (dec *Decoder) recvType(id typeId) {
        // Have we already seen this type? That's an error
-       if id < firstUserId || dec.wireType[id] != nil {
+       if id < firstUserId /*|| dec.wireType[id] != nil*/ {
                dec.err = errors.New("gob: duplicate type received")
                return
        }

and does yield the type of performance improvement that I'm looking for:

BenchmarkDecode/WithTypes-8                55258             21711 ns/op
BenchmarkDecode/WithoutTypes-8            607665              2002 ns/op
BenchmarkDecode/WithReuse-8               269878              4509 ns/op

However I'm not sure how best to do this safely.

Since type ids are generated on demand, and values are only sent when non-zero, it seems that two objects would have overlapping type ids assigned if different fields were empty. Is that correct?

In that case, the id alone would not be enough and the patch would need to be smarter like so:

--- a/src/encoding/gob/decoder.go
+++ b/src/encoding/gob/decoder.go
@@ -57,7 +57,7 @@ func NewDecoder(r io.Reader) *Decoder {
 // recvType loads the definition of a type.
 func (dec *Decoder) recvType(id typeId) {
        // Have we already seen this type? That's an error
-       if id < firstUserId || dec.wireType[id] != nil {
+       if id < firstUserId {
                dec.err = errors.New("gob: duplicate type received")
                return
        }
@@ -68,6 +68,12 @@ func (dec *Decoder) recvType(id typeId) {
        if dec.err != nil {
                return
        }
+       if old := dec.wireType[id]; old != nil {
+               if same(old, wire) {
+                       // Same as old type, ignore and use existing
+                       return
+               }
+       }
        // Remember we've seen this type.
        dec.wireType[id] = wire
 }

Would something like this be feasible, or is there other state inside the decoder that would also need to be updated to make this work reliably?

Since type ids are generated on demand, and values are only sent when non-zero, it seems that two objects would have overlapping type ids assigned if different fields were empty. Is that correct?

This assumption appears to be incorrect. From what I can see, type information is sent for every nested type regardless of value. If that's true in all cases, then it suggests that type identifiers would remain stable across encoder invocations as long as the underlying type has the same fields in the same order.

If type ids are in fact stable in this case, then it becomes much safer to re-use decoder instances with the patch above. The same() check in the patch can become a simple (*wireType).string() comparison.

If gob.Decoder had a mode where it ignored the "duplicate type received" errors, and only complained if the underlying type also changed, that would be sufficient for my needs. As an API user, I could cache and re-use a Decoder instance per type. With a new "conflicting type id received" error I could detect and invalidate my cached Decoder in the rare case where it tried to parse two different generations of values sequentially.

Does this seem like a reasonable approach?

If type ids are in fact stable in this case, then it becomes much safer to re-use decoder instances with the patch above. The same() check in the patch can become a simple (*wireType).string() comparison.

While this is a reasonable heuristic, it is not perfect. Specifically, (*wireType).string() only refers to the type's name, so the dec.wireType map looks something like:

65 = obj
66 = map[string]interface {}
67 = []string
68 = []float64

By checking to ensure the same type name and id are received, we can detect when fields are added or reordered, but only as long as their types also change.

So if someone simply reordered two fields with the same type (i.e []string), re-using a decoder instance with the simplistic patch above would result in the values being deserialized into the wrong place.

Perhaps a better checksum can be computed elsewhere containing the names of the fields as well, to more effectively error out when the underlying structure differs from the compiled decoder?

Specifically, (*wireType).string() only refers to the type's name

Instead of using this for the sameness check, I tried to use reflect.DeepEqual and confirmed that would properly detect field reordering within struct types. However the performance of reflection here was quite bad, so I implemented a simple same() instead.

Below is an updated version of my patch, which allows me to reuse a single decoder instance across streams. An error is only thrown in the case where the same decoder is used to try to decode two different generations of an object, signaling to the caller that the fields have been altered in a way that the cached decoder needs to be recreated and retried.

The performance is 4-5x better compared to creating a fresh decoder every time.

If this new behavior makes sense and is worth including in stdlib (optionally under a new setting), then I can take some time to write tests and submit a CL. But if it's not likely to be accepted then I will just maintain a fork.

BenchmarkDecode/WithTypes-8         	   38244	     31949 ns/op
BenchmarkDecode/WithoutTypes-8      	  441136	      2676 ns/op
BenchmarkDecode/WithReuse-8         	  174122	      6856 ns/op

diff --git a/src/encoding/gob/decoder.go b/src/encoding/gob/decoder.go
index b52aabe54b..5957a7435d 100644
--- a/src/encoding/gob/decoder.go
+++ b/src/encoding/gob/decoder.go
@@ -56,9 +56,9 @@ func NewDecoder(r io.Reader) *Decoder {
 
 // recvType loads the definition of a type.
 func (dec *Decoder) recvType(id typeId) {
-	// Have we already seen this type? That's an error
-	if id < firstUserId || dec.wireType[id] != nil {
-		dec.err = errors.New("gob: duplicate type received")
+	// Is this type id within allowed range?
+	if id < firstUserId {
+		dec.err = errors.New("gob: reserved type id received")
 		return
 	}
 
@@ -68,6 +68,13 @@ func (dec *Decoder) recvType(id typeId) {
 	if dec.err != nil {
 		return
 	}
+
+	// Have we already seen this type id for a different type? That's an error
+	if old := dec.wireType[id]; old != nil && !old.same(wire) {
+		dec.err = errors.New("gob: conflicting type id received")
+		return
+	}
+
 	// Remember we've seen this type.
 	dec.wireType[id] = wire
 }
diff --git a/src/encoding/gob/type.go b/src/encoding/gob/type.go
index 31c0ef7af1..a41c90c99d 100644
--- a/src/encoding/gob/type.go
+++ b/src/encoding/gob/type.go
@@ -670,6 +670,53 @@ func (w *wireType) string() string {
 	return unknown
 }
 
+func (w *wireType) same(v *wireType) bool {
+	if w == v {
+		return true
+	}
+	switch {
+	case w.ArrayT != nil &&
+		v.ArrayT != nil:
+		return w.ArrayT.Name == v.ArrayT.Name &&
+			w.ArrayT.Elem == v.ArrayT.Elem &&
+			w.ArrayT.Len == v.ArrayT.Len
+	case w.SliceT != nil &&
+		v.SliceT != nil:
+		return w.SliceT.Name == v.SliceT.Name &&
+			w.SliceT.Elem == v.SliceT.Elem
+	case w.StructT != nil &&
+		v.StructT != nil:
+		if w.StructT.Name != v.StructT.Name {
+			return false
+		}
+		if len(w.StructT.Field) != len(v.StructT.Field) {
+			return false
+		}
+		for i := range w.StructT.Field {
+			if w.StructT.Field[i].Name != v.StructT.Field[i].Name ||
+				w.StructT.Field[i].Id != v.StructT.Field[i].Id {
+				return false
+			}
+		}
+		return true
+	case w.MapT != nil &&
+		v.MapT != nil:
+		return w.MapT.Name == v.MapT.Name &&
+			w.MapT.Key == v.MapT.Key &&
+			w.MapT.Elem == v.MapT.Elem
+	case w.GobEncoderT != nil &&
+		v.GobEncoderT != nil:
+		return w.GobEncoderT.Name == v.GobEncoderT.Name
+	case w.BinaryMarshalerT != nil &&
+		v.BinaryMarshalerT != nil:
+		return w.BinaryMarshalerT.Name == v.BinaryMarshalerT.Name
+	case w.TextMarshalerT != nil &&
+		v.TextMarshalerT != nil:
+		return w.TextMarshalerT.Name == v.TextMarshalerT.Name
+	}
+	return false
+}
+
 type typeInfo struct {
 	id      typeId
 	encInit sync.Mutex   // protects creation of encoder

FYI, wiring up this cacheable decoder to my app resulted in a 3x performance improvement in the gob unmarshal path:

BenchmarkUnmarshal/WithoutCache-8         	    9024	    132112 ns/op
BenchmarkUnmarshal/WithCache-8            	   27282	     43448 ns/op

This is with a fairly complex struct with lots of nested types.

Though things looked great in my app benchmarks, with a real world dataset there are enough variations across object generations that many of the advantages of caching are lost.

I think it's fair to say that the design of encoding/gob makes it unsuitable for my use-case.