New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
affected/package: crypto AEAD modes #59659
Comments
That's a limitation of your hardware, not the code |
Your code never reaches the crypto part before it gets killed. Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only. For questions please refer to https://github.com/golang/go/wiki/Questions |
@seankhliao, it's not just hardware limitation. It is perfectly possible to encrypt 2GB in CBC or CTR mode, the error only occurs with AEAD modes. It makes no sense that a machine with 3GB of RAM cannot encrypt more than 250MB in AEAD mode. This is not a hardware limitation, but a software one. The proof of this is that other modes are capable of encrypting large files.
The only thing I was told there was to split the file into pieces, encrypt them separately and merge them, which, by itself, is not a solution as each chunk would have a different tag. |
@seankhliao, I wrote: buf := bytes.NewBuffer(nil)
var data io.Reader
var size int64
data = os.Stdin
fi, err := os.Stdin.Stat()
if err != nil {
log.Fatal(err)
}
size = fi.Size()
buf.Grow(int(size))
io.Copy(buf, data)
msg := buf.Bytes() But this results in ciphertext twice the size of plaintext on 64-bit machines. And it also doesn't solve the problem of files larger than 250mb. |
streaming aead is #43774 |
What version of Go are you using (
go version
)?1.20
Does this issue reproduce with the latest release?
Yep
What operating system and processor architecture are you using (
go env
)?Windows and Linux
What did you do?
What did you expect to see?
Based on NIST SP 800-38D (GCM) section 5.2.1.1, it seems that the maximum length of plaintext is 2^39-256 bits ~ 64 GB.
What did you see instead?
Files larger than
1.5GB250MB will result in a memory error with any AEAD operating mode, it is not possible to encrypt1.5GB250MB files:runtime: out of memory: cannot allocate 1073741824-byte block (1077837824 in use)
fatal error: out of memory
runtime stack:
runtime.throw(0x4d9c26, 0xd)
c:/go/src/runtime/panic.go:617 +0x64
runtime.largeAlloc(0x3ffffe00, 0x11c10101, 0x11c1e000)
c:/go/src/runtime/malloc.go:1057 +0x10f
runtime.mallocgc.func1()
c:/go/src/runtime/malloc.go:950 +0x39
runtime.systemstack(0x447245)
c:/go/src/runtime/asm_386.s:396 +0x53
runtime.mstart()
c:/go/src/runtime/proc.go:1153
goroutine 1 [running]:
runtime.systemstack_switch()
c:/go/src/runtime/asm_386.s:357 fp=0x11c56d94 sp=0x11c56d90 pc=0x447300
runtime.mallocgc(0x3ffffe00, 0x4c0340, 0x1, 0x11c56e00)
c:/go/src/runtime/malloc.go:949 +0x65b fp=0x11c56de8 sp=0x11c56d94 pc=0x40968b
runtime.makeslice(0x4c0340, 0x3ffffe00, 0x3ffffe00, 0x527ffe00)
c:/go/src/runtime/slice.go:49 +0x4f fp=0x11c56dfc sp=0x11c56de8 pc=0x43521f
bytes.makeSlice(0x3ffffe00, 0x0, 0x0, 0x0)
c:/go/src/bytes/buffer.go:232 +0x61 fp=0x11c56e10 sp=0x11c56dfc pc=0x491821
bytes.(*Buffer).grow(0x11c3bb60, 0x200, 0x10000000)
c:/go/src/bytes/buffer.go:145 +0x12a fp=0x11c56e38 sp=0x11c56e10 pc=0x4913ca
bytes.(*Buffer).ReadFrom(0x11c3bb60, 0x4f5920, 0x11c380d8, 0x3880e0, 0x11c3bb60, 0x1, 0x75)
c:/go/src/bytes/buffer.go:205 +0x45 fp=0x11c56e74 sp=0x11c56e38 pc=0x491645
io.copyBuffer(0x4f5880, 0x11c3bb60, 0x4f5920, 0x11c380d8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4d8408, ...)
c:/go/src/io/io.go:388 +0x29a fp=0x11c56eb4 sp=0x11c56e74 pc=0x44e65a
io.Copy(...)
c:/go/src/io/io.go:364
main.main()
H:/PGMM/crypter/aes-gcm/main.go:93 +0x5c6 fp=0x11c56fd0 sp=0x11c56eb4 pc=0x4a9366
runtime.main()
c:/go/src/runtime/proc.go:200 +0x1d7 fp=0x11c56ff0 sp=0x11c56fd0 pc=0x427937
runtime.goexit()
c:/go/src/runtime/asm_386.s:1321 +0x1 fp=0x11c56ff4 sp=0x11c56ff0 pc=0x448b21
The text was updated successfully, but these errors were encountered: