You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The repro case below creates a map that's a few GB big and then attempts to encode it to a file using Encode(). Encode() fails with a "message too big" error. (Presumably because the full size of the map is greater than 1<<30, which is the value of "tooBig" in the gob implementation.
I would expect the gob implementation to be able to handle a map that's larger than the maximum message size, since the individual elements being encoded are much smaller than the limit.
package main
import (
"encoding/gob"
"fmt"
"io/ioutil"
"log"
)
type File struct {
A, B string
V int
}
func main() {
f, err := ioutil.TempFile("", "foo")
if err != nil {
panic(err)
}
e := gob.NewEncoder(f)
m := make(map[string][]*File)
for i := 0; i < (1 << 22); i += 1 {
fl := File{"00000000000000000000000000000000000000000000000",
"11111111111111111111111111111111111111111", 42}
var arr []*File
for j := 0; j < 4; j += 1 {
arr = append(arr, &fl)
}
m[fmt.Sprintf("%d", i)] = arr
if i%(1<<20) == 0 {
log.Printf("i = %d / %d", i, 1<<22)
}
}
if err := e.Encode(m); err != nil {
panic(err)
}
}
The text was updated successfully, but these errors were encountered:
ianlancetaylor
changed the title
gob Encode() gives "encoder: message too big" when given a large map
encoding/gob: Encode gives "encoder: message too big" when given a large map
Jan 7, 2016
mmp
pushed a commit
to google/skicka
that referenced
this issue
Jan 8, 2016
Serialize the file id to File struct map manually, rather than by passing it to gob/Encode() in its entirety, as the Encode() implementation recently changed to issue an error if the map is very large.
(I believe that this will fix skicka issue #98.)
Sorry but no. The model requires that a data item is transmitted atomically (although types may break the stream into submessages). Too hard to change.
OSX / go version go1.5.1 darwin/amd64
The repro case below creates a map that's a few GB big and then attempts to encode it to a file using Encode(). Encode() fails with a "message too big" error. (Presumably because the full size of the map is greater than 1<<30, which is the value of "tooBig" in the gob implementation.
I would expect the gob implementation to be able to handle a map that's larger than the maximum message size, since the individual elements being encoded are much smaller than the limit.
The text was updated successfully, but these errors were encountered: