encoding/gob: tooBig is too small #27635

bf · 2018-09-12T06:47:30Z

What version of Go are you using (`go version`)?

go version go1.10.3 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (`go env`)?

arch linux with 16gb ram

What did you do?

I parse large CSV files for social media analytics, creating in-memory data structures from the data. When persisting these data structures to disk using gob, I receive gob: encoder: message too big error from this code:

encoder := gob.NewEncoder(file)
err := encoder.Encode(MapData)

One set of input data is a 4GB csv file which results in a data structure that is 200MB when saved with gob. The second set of input data is a 15GB csv file which should result in a 1200MB file when saved with gob, but I receive the error above.

The error is thrown at a check if the data structure is larger than the tooBig constant at

go/src/encoding/gob/encoder.go

Line 70 in 50bd1c4

if messageLen >= tooBig {

The tooBig constant is originally defined with a comment TODO: Make this adjustable here

go/src/encoding/gob/decoder.go

Line 17 in 50bd1c4

// TODO: make this adjustable?

go/src/encoding/gob/decoder.go

Line 18 in 50bd1c4

const tooBig = 1 << 30

What did you expect to see?

I expect gob to save large data structures of ~2GB that dont get close to my RAM limits of 16GB without any problems.

What did you see instead?

I got the gob: encoder: message too big error.

I fixed the error by compiling my own go and changing the tooBig check in encoder.go to 1<<31. Afterwards the data structure from the large 15GB CSV file was successfully saved by gob in a 1500MB gob file.

Therefore I suggest this change could be done for all users, as the existing TODO comment in the golang source code suggests

go/src/encoding/gob/decoder.go

Line 17 in 50bd1c4

// TODO: make this adjustable?

The text was updated successfully, but these errors were encountered:

agnivade · 2018-09-12T07:00:46Z

/cc @robpike

mvdan · 2018-09-12T07:13:17Z

This was first introduced in 9c3fc83, because earlier it was easy to get encoding/gob to panic with large amounts of data due to overflows.

My initial reaction would be "we can just bump this limit on 64-bit", or "we can just use 64-bit sized integers in the encoder and decoder", although I don't know the internals of the gob package that well.

The original issue was #8084. Interestingly enough, it happened on amd64.

I guess it's fine to double the limit while we don't have a better long-term solution, but I'll leave that to Rob to decide.

bf · 2018-09-12T10:36:11Z

@mvdan thanks for your input. I tried to looking through the gob source but I can't really make out what kind of integers are used.

But what I can tell you is that I changed the data structure from map[int]map[int]map[int]map[uint64]bool to map[int8]map[int8]map[int8]map[uint64]bool in order to save disk space and potentially work around the error message. I found out the size of the gob dump file did not change when using int8 instead of the standard int (which afaik is at least int32).

magical · 2018-09-13T04:36:32Z

@bf FYI, gob uses a variable-length format to encode integers, so it makes sense that changing the integer type in your data structure did not affect the output size.

rsc · 2018-09-19T18:15:08Z

Maybe tooBig should be bigger on 64-bit. On 32-bit I think 1GB is fine. Thoughts, @robpike?

robpike · 2018-09-20T02:54:45Z

Sure.

gopherbot · 2018-10-22T03:18:39Z