Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

affected/package: crypto AEAD modes #59659

Closed
pedroalbanese opened this issue Apr 16, 2023 · 5 comments
Closed

affected/package: crypto AEAD modes #59659

pedroalbanese opened this issue Apr 16, 2023 · 5 comments

Comments

@pedroalbanese
Copy link

pedroalbanese commented Apr 16, 2023

What version of Go are you using (go version)?

1.20

Does this issue reproduce with the latest release?

Yep

What operating system and processor architecture are you using (go env)?

Windows and Linux

What did you do?

package main
import (
	"crypto/cipher"
	"crypto/rand"
	"crypto/aes"
	"bytes"
	"encoding/hex"
	"flag"
	"fmt"
	"crypto/sha256"
	"golang.org/x/crypto/pbkdf2"
	"io"
	"log"
	"os"
)

	var dec = flag.Bool("d", false, "Decrypt instead Encrypt.")
	var iter = flag.Int("i", 1024, "Iterations. (for PBKDF2)")
	var key = flag.String("k", "", "128-bit key to Encrypt/Decrypt.")
	var pbkdf = flag.String("p", "", "PBKDF2.")
	var salt = flag.String("s", "", "Salt. (for PBKDF2)")

func main() {
    flag.Parse()

        if (len(os.Args) < 1) {
	fmt.Println("Usage of",os.Args[0]+":")
        flag.PrintDefaults()
        os.Exit(1)
        }
	
	var keyHex string
	var prvRaw []byte
	if *pbkdf != "" {
	prvRaw = pbkdf2.Key([]byte(*pbkdf), []byte(*salt), *iter, 32, sha256.New)
	keyHex = hex.EncodeToString(prvRaw)
	} else {
	keyHex = *key
	}
	var key []byte
	var err error
	if keyHex == "" {
		key = make([]byte, 32)
		_, err = io.ReadFull(rand.Reader, key)
		if err != nil {
                        log.Fatal(err)
		}
		fmt.Fprintln(os.Stderr, "Key=", hex.EncodeToString(key))
	} else {
		key, err = hex.DecodeString(keyHex)
		if err != nil {
                        log.Fatal(err)
		}
		if len(key) != 32 {
                        log.Fatal(err)
		}
	}

	block, err := aes.NewCipher(key)
	if err != nil {
		panic(err.Error())
	}

	aead, err := cipher.NewGCM(block)
	if err != nil {
		panic(err.Error())
	}

	if *dec == false {
		buf := bytes.NewBuffer(nil)
		data := os.Stdin 
		io.Copy(buf, data)
		msg := buf.Bytes()

		nonce := make([]byte, aead.NonceSize(), aead.NonceSize()+len(msg)+aead.Overhead())

		out := aead.Seal(nonce, nonce, msg, nil)
		fmt.Printf("%s", out)

	        os.Exit(0)
	}

	if *dec == true {
		buf := bytes.NewBuffer(nil)
		data := os.Stdin 
		io.Copy(buf, data)
		msg := buf.Bytes()

		nonce, msg := msg[:aead.NonceSize()], msg[aead.NonceSize():]

		out, err := aead.Open(nil, nonce, msg, nil)
		if err != nil {
			panic(err)
		}
		fmt.Printf("%s", out)

	        os.Exit(0)
	}
}

What did you expect to see?

Based on NIST SP 800-38D (GCM) section 5.2.1.1, it seems that the maximum length of plaintext is 2^39-256 bits ~ 64 GB.

What did you see instead?

Files larger than 1.5GB 250MB will result in a memory error with any AEAD operating mode, it is not possible to encrypt 1.5GB 250MB files:

runtime: out of memory: cannot allocate 1073741824-byte block (1077837824 in use)
fatal error: out of memory

runtime stack:
runtime.throw(0x4d9c26, 0xd)
c:/go/src/runtime/panic.go:617 +0x64
runtime.largeAlloc(0x3ffffe00, 0x11c10101, 0x11c1e000)
c:/go/src/runtime/malloc.go:1057 +0x10f
runtime.mallocgc.func1()
c:/go/src/runtime/malloc.go:950 +0x39
runtime.systemstack(0x447245)
c:/go/src/runtime/asm_386.s:396 +0x53
runtime.mstart()
c:/go/src/runtime/proc.go:1153

goroutine 1 [running]:
runtime.systemstack_switch()
c:/go/src/runtime/asm_386.s:357 fp=0x11c56d94 sp=0x11c56d90 pc=0x447300
runtime.mallocgc(0x3ffffe00, 0x4c0340, 0x1, 0x11c56e00)
c:/go/src/runtime/malloc.go:949 +0x65b fp=0x11c56de8 sp=0x11c56d94 pc=0x40968b
runtime.makeslice(0x4c0340, 0x3ffffe00, 0x3ffffe00, 0x527ffe00)
c:/go/src/runtime/slice.go:49 +0x4f fp=0x11c56dfc sp=0x11c56de8 pc=0x43521f
bytes.makeSlice(0x3ffffe00, 0x0, 0x0, 0x0)
c:/go/src/bytes/buffer.go:232 +0x61 fp=0x11c56e10 sp=0x11c56dfc pc=0x491821
bytes.(*Buffer).grow(0x11c3bb60, 0x200, 0x10000000)
c:/go/src/bytes/buffer.go:145 +0x12a fp=0x11c56e38 sp=0x11c56e10 pc=0x4913ca
bytes.(*Buffer).ReadFrom(0x11c3bb60, 0x4f5920, 0x11c380d8, 0x3880e0, 0x11c3bb60, 0x1, 0x75)
c:/go/src/bytes/buffer.go:205 +0x45 fp=0x11c56e74 sp=0x11c56e38 pc=0x491645
io.copyBuffer(0x4f5880, 0x11c3bb60, 0x4f5920, 0x11c380d8, 0x0, 0x0, 0x0, 0x0, 0x0, 0x4d8408, ...)
c:/go/src/io/io.go:388 +0x29a fp=0x11c56eb4 sp=0x11c56e74 pc=0x44e65a
io.Copy(...)
c:/go/src/io/io.go:364
main.main()
H:/PGMM/crypter/aes-gcm/main.go:93 +0x5c6 fp=0x11c56fd0 sp=0x11c56eb4 pc=0x4a9366
runtime.main()
c:/go/src/runtime/proc.go:200 +0x1d7 fp=0x11c56ff0 sp=0x11c56fd0 pc=0x427937
runtime.goexit()
c:/go/src/runtime/asm_386.s:1321 +0x1 fp=0x11c56ff4 sp=0x11c56ff0 pc=0x448b21

@pedroalbanese pedroalbanese changed the title affected/package: affected/package: AEAD modes Apr 16, 2023
@pedroalbanese pedroalbanese changed the title affected/package: AEAD modes affected/package: crypto AEAD modes Apr 16, 2023
@seankhliao
Copy link
Member

That's a limitation of your hardware, not the code

@seankhliao seankhliao closed this as not planned Won't fix, can't repro, duplicate, stale Apr 16, 2023
@seankhliao
Copy link
Member

Your code never reaches the crypto part before it gets killed.
The inefficient use of bytes.Buffer is what uses the memory.
Adding a line like buf.Grow(3 * 1024 * 1024 * 1024) lets it handle a ~3GB file.

Unlike many projects, the Go project does not use GitHub Issues for general discussion or asking questions. GitHub Issues are used for tracking bugs and proposals only.

For questions please refer to https://github.com/golang/go/wiki/Questions

@pedroalbanese
Copy link
Author

@seankhliao, it's not just hardware limitation. It is perfectly possible to encrypt 2GB in CBC or CTR mode, the error only occurs with AEAD modes. It makes no sense that a machine with 3GB of RAM cannot encrypt more than 250MB in AEAD mode. This is not a hardware limitation, but a software one. The proof of this is that other modes are capable of encrypting large files.

For questions please refer to https://github.com/golang/go/wiki/Questions

The only thing I was told there was to split the file into pieces, encrypt them separately and merge them, which, by itself, is not a solution as each chunk would have a different tag.

@pedroalbanese
Copy link
Author

pedroalbanese commented Apr 18, 2023

@seankhliao, buf.Grow(3 * 1024 * 1024 * 1024) exceeds the eighth Mersenne prime making it impossible on 32-bit machines as RaspberryPi armel.

I wrote:

	buf := bytes.NewBuffer(nil)
	var data io.Reader
	var size int64
	data = os.Stdin
	fi, err := os.Stdin.Stat()
	if err != nil {
		log.Fatal(err)
	}
	size = fi.Size()
	buf.Grow(int(size))
	io.Copy(buf, data)
	msg := buf.Bytes()

But this results in ciphertext twice the size of plaintext on 64-bit machines. And it also doesn't solve the problem of files larger than 250mb.

@seankhliao
Copy link
Member

streaming aead is #43774

@golang golang locked and limited conversation to collaborators Apr 17, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants