/cc @agl

Quoting @agl from 2013 regarding this (https://groups.google.com/forum/#!topic/golang-nuts/AhNnnzLMUIE)

"It just refers to the fact that the API is in-memory, rather than streaming. So if you needed to encrypt something larger than half the size of memory you would either need to encrypt in chunks or do a bunch of work to make a streaming version of the code."

@mdp what would happen if you tried to encrypt something larger than memory/2? The function doesn't return an error, so I presume the system would just crash? When testing, I managed to encrypt fairly large amounts of data relatively quickly, although I don't remember how much data exactly, just that it was in the GBs.

It doesn't seem like a very portable system, since you wouldn't really know the amount of available memory on the target system. I think the best solution would be to pick something very small like a few MB and code a wrapper that'd encrypt in chunks, but then you'd be storing a nonce for every single chunk and it could easily get messy.

Someone please correct me if I'm missing something.

I assume you'll eventually hit one of two limits, physical memory + swap, or the Golang hard limit of 128gb.

Your solution to break it up into chunks seems like the best route to me. Streaming ciphers are another option, but they're not easy to get right (https://leanpub.com/gocrypto/read#leanpub-auto-messages-v-streams)

As for keeping track of the nonce, you could just generate a single nonce and then increment it for each additional chunk. According to Libsodium, which is based on 'nacl' - "The public nonce should never ever be reused with the same key. The recommended way to generate it is to use SecretAead.GenerateNonce() for the first message, and increment it for each subsequent message using the same key."

@mdp yeah for the current project I'm working on it's not a problem since I know exactly how big the plain texts are. However in the future I'll have to try and implement something along the lines of what you described.

@agl 'box' is also implemented in-memory, but doesn't have a warning. Maybe it would make sense to add a small warning to both docs along the lines of: "[secret]box API operates in-memory, therefore message size should be limited"

@mdp that sounds like a good idea. Maybe elaborate somewhat further though to make it more clear.

Something like "X is implemented in-memory, and therefore message-sizes are limited by the amount of RAM on the target system. This should be kept in mind when using the API." If there's a hard-limit then maybe that should be included too, i.e. N/2 was mentioned earlier.

Also, would it be feasible to check the size of the plaintext on runtime and compare it with the amount of memory available on the system? I feel that it could be possible, but I'm leaning towards favouring allowing developers to choose to implement that themselves.

There's also a deeper reason why this API is in-memory only and that it's good to only encrypt small messages. In short, old standards (e.g. PGP) will encrypt plaintexts of any length and then put an authenticator at the end. The likely outcome of such a design is that some implementations will stream out unauthenticated plaintext and only notice any problem when they get to the end of the ciphertext and try to check the authenticator. But by that time the damage has been done—it doesn't take much searching to find people suggesting piping the output of gpg to tar or even a shell.

So encrypting large messages (i.e. > 16KB, or even less depend on how small implementations might be) pressures people to implement dangerous designs like the above. Thus large messages should be chunked.

Streaming decryption always runs the risk of an attacker truncating the ciphertext however. Designs must always be able to detect truncation, but it's very easy to imagine that the rest of a system doesn't handle it well. (For example, see the Cookie Cutter attack against HTTPS.)

(I've rambled at greater length about this here.)

I'll see if I can amend that comment to make it more precise without making it too long. (And it should, indeed, be replicated in box because the considerations are the same.)

Damn, @agl is the Adam Langley! I never realised. I love your blog.

Cough back on topic: I agree that amending the documentation is the best solution. I hadn't previously considered the implications of streaming encryption, and so doing it in chunks is definitely the better option.

This should be made clear somehow though, because not everyone using the NaCl library will necessarily know that encrypting huge plaintexts is bad. The documentation should mention the security concerns as well as just the memory warning.

I made an attempt at this in https://go-review.googlesource.com/35910. I'll have to see whether I still think it's reasonable tomorrow though.

CL https://golang.org/cl/35910 mentions this issue.