This is sort of expected; the current limit is

// tooBig provides a sanity check for sizes; used in several places. Upper limit
// of is 1GB on 32-bit systems, 8GB on 64-bit, allowing room to grow a little
// without overflow.
const tooBig = (1 << 30) << (^uint(0) >> 62)

It used to be 1GiB on 64-bit systems, but got raised to 8GiB per #27635. It is a hard-coded limit, so nothing will really be high enough and low enough for everyone.

One potential middle ground to satisfy everyone could be to not allocate the entire buffer upfront if its size is huge - say, above 128MiB. In those "huge buffer" scenarios, we would let the buffer grow as the contents get read in, starting with a reasonably small buffer. This way we would still support content that is up to 8GiB in size, but we wouldn't allocate up to 8GiB with a <1KiB input payload, which is an extremely cheap OOM/crash attack. One could still make a Go program run out of memory with this strategy, but it would likely take gigabytes of input to actually do so, making the attack much less interesting in the face of network speeds, rate limits, etc.

That strategy would make real large inputs somewhat slower, as we would repeatedly grow and copy the buffer, but I think that's probably an OK tradeoff if the growth is fast enough, akin to append.

cc @robpike @bf @rsc and potentially @golang/security for visibility

One potential middle ground to satisfy everyone could be to not allocate the entire buffer upfront if its size huge

Indeed, that is what is being done with saferio cf https://go-review.googlesource.com/c/go/+/412014/ for instance

Ah, so my idea above isn't all that creative :) I think doing the same in gob is very reasonable.

Not sure how allocating lazily helps reduce memory load. The decoder has the size of the data item. Assuming the size is correct, the right thing to do is to allocate the buffer and just do the read. It's simple and fast. Not only that, allocating piecemeal will result in more memory being allocated, not less.

Encoding/gob may not be the most robust system, it expects the data to be honest. Protecting against lies in the stream can only get you so far.

Fuzzing bugs are often like this, throwing random noise at things to see if things can break. For a data stream decoder capable of reading massive data items, it's easy to trigger an OOM and not at all clear how to prevent them. The feeble attempt involving that constant is the best we've come up with.

Addendum: Gob is best used to stream data between related, known systems. It is not the best choice when accepting data from untrusted sources.

Not sure how allocating lazily helps reduce memory load.

The fuzzer/attacker needs to produce 1Gbyte of data (instead of a few bytes) to get gob to allocate 1Gbyte

I am ok if this is the expected behavior as gob should not come from untrusted sources (as for index/sufixarray)

Another fix could be to let the user specify the tooBig value...

allocating piecemeal will result in more memory being allocated, not less.

As far as OOMs go, I think we're worried about peak memory usage. For the case of a truly large input, growing the buffer in small steps is presumably not significantly worse than allocating the entire buffer at once, as we can rely on the GC to get rid of the older buffers. And if the user really wants to support reading huge inputs, I imagine they want to ensure they have plenty of memory available anyway - or otherwise, redesign their code to split the gob into smaller messages.

I also don't think that this will affect the vast majority of gob users, as they are hopefully not encoding gigabytes of data as a single message. saferio allows starting with a 10MiB buffer, for example, but we could allow gob to start with a larger buffer if we think that would be more appropriate.

it's easy to trigger an OOM

I generally disagree on your idea of "easy" here :) If I wanted to crash a website with this attack as it stands now, I would only have to send less than 1KiB of data. With the proposed fix, I would have to send many gigabytes - on my ~7Mb upload connection, sending 8GiB would take upwards of two hours, assuming it doesn't fail. I could possibly rent a server somewhere with a faster connection, but they will usually charge you for using large amounts of bandwidth.

And all of this assumes I would succeed at uploading 8GiB to a server without that failing somehow. If we talk about HTTP endpoints, for example, I've found it's pretty common to have proxies which limit the size of requests or rate limit the amount of data that a single client can send in a short period of time, either of which would make the proposed fix pretty effective.

Change https://go.dev/cl/413979 mentions this issue: encoding/gob: use saferio to read large buffer