Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

math/big: panic: mismatched montgomery number lengths #48431

Closed
fisherFIAF opened this issue Sep 17, 2021 · 11 comments
Closed

math/big: panic: mismatched montgomery number lengths #48431

fisherFIAF opened this issue Sep 17, 2021 · 11 comments
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@fisherFIAF
Copy link

fisherFIAF commented Sep 17, 2021 •
edited

go version 1.16.6

GRPC panicked when serving.

tracing stack:

panic: math/big: mismatched montgomery number lengths

goroutine 1787827 [running]:
math/big.nat.montgomery(0x0, 0x0, 0x0, 0xc0056b8f80, 0x10, 0x10, 0xc0056b7400, 0x1a, 0x26, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:219 +0x507
math/big.nat.expNNMontgomery(0xc0035926c0, 0x40, 0x44, 0xc003592b40, 0x10, 0x45, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1430 +0x327
math/big.nat.expNN(0xc0035926c0, 0x40, 0x44, 0xc003592000, 0x40, 0x44, 0xc000478a00, 0x10, 0x25, 0xc0008fe0b0, ...)
        /usr/local/go/src/math/big/nat.go:1262 +0xa06
math/big.(*Int).Exp(0xc003f733d8, 0xc003f733f8, 0xc0005ca3e0, 0xc0005ca340, 0xc000bbbf60)
        /usr/local/go/src/math/big/int.go:509 +0x1a5
crypto/rsa.decrypt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f733f8, 0xc003f734e8, 0x5d4506, 0x13fd3c0)
        /usr/local/go/src/crypto/rsa/rsa.go:536 +0x19f
crypto/rsa.decryptAndCheck(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0xc003f73580, 0x100, 0x100, 0xc0053d57a0)
        /usr/local/go/src/crypto/rsa/rsa.go:570 +0x53
crypto/rsa.signPSSWithSalt(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0019e4ae0, 0x20, 0x20, ...)
        /usr/local/go/src/crypto/rsa/pss.go:217 +0x1c5
crypto/rsa.SignPSS(0x18a0900, 0xc0000e8480, 0xc0005d26c0, 0x5, 0xc0019e4ac0, 0x20, 0x20, 0xc0097e5fe0, 0x40df45, 0x14b45e0, ...)
        /usr/local/go/src/crypto/rsa/pss.go:281 +0x1da
crypto/rsa.(*PrivateKey).Sign(0xc0005d26c0, 0x18a0900, 0xc0000e8480, 0xc0019e4ac0, 0x20, 0x20, 0x18a0920, 0xc0097e5fe0, 0xc005580000, 0x25, ...)
        /usr/local/go/src/crypto/rsa/rsa.go:146 +0x9e
crypto/tls.(*serverHandshakeStateTLS13).sendServerCertificate(0xc003f739e8, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:636 +0x466
crypto/tls.(*serverHandshakeStateTLS13).handshake(0xc003f739e8, 0xc00103ac00, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server_tls13.go:59 +0xd4
crypto/tls.(*Conn).serverHandshake(0xc005ab3c00, 0x0, 0x7f982045c2c0)
        /usr/local/go/src/crypto/tls/handshake_server.go:51 +0xd4
crypto/tls.(*Conn).Handshake(0xc005ab3c00, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1391 +0xc9
google.golang.org/grpc/credentials.(*tlsCreds).ServerHandshake(0xc00063c230, 0x18d9ee0, 0xc001212180, 0x244b200, 0xe9, 0x0, 0x0, 0xc0005176f0, 0x78)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/credentials/tls.go:115 +0xdf
google.golang.org/grpc.(*Server).useTransportAuthenticator(0xc000819500, 0x18d9ee0, 0xc001212180, 0x244b200, 0x0, 0x0, 0x244b200, 0xc0058ffbf0, 0xc0492d074d535ba5)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:651 +0x50
google.golang.org/grpc.(*Server).handleRawConn(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:776 +0xca
google.golang.org/grpc.(*Server).Serve.func3(0xc000819500, 0x18d9ee0, 0xc001212180)
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:762 +0x3f
created by google.golang.org/grpc.(*Server).Serve
        /var/lib/jenkins/workspace/release/server/pkg/mod/google.golang.org/grpc@v1.32.0/server.go:761 +0x305

It happened when server is online, and very seldom, nearly once one or two weeks.
It seems happened when GRPC server handshake when new connection come.

linux kernel version: 3.10.0-1160.36.2.el7.x86_64

CPU:

processor       : 31
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6151 CPU @ 3.00GHz
stepping        : 4
microcode       : 0x1
cpu MHz         : 3000.000
cache size      : 25344 KB
physical id     : 1
siblings        : 16
core id         : 7
cpu cores       : 8
apicid          : 31
initial apicid  : 31
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl xtopology nonstop_tsc eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx avx512f avx512dq rdseed adx smap clflushopt clwb avx512cd avx512bw avx512vl xsaveopt xsavec xgetbv1 arat md_clear spec_ctrl intel_stibp flush_l1d
bogomips        : 6000.00
clflush size    : 64
cache_alignment : 64
address sizes   : 42 bits physical, 48 bits virtual
power management:
@ericlagergren
Copy link
Contributor

ericlagergren commented Sep 17, 2021 •
edited

This looks like a security bug: https://golang.org/security

@FiloSottile

@randall77
Copy link
Contributor

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

@fisherFIAF
Copy link
Author

I'm not sure how this could happen. The RR variable has 26 words instead of 16. That could only happen if div is somehow broken.

can I resolve this problem by upgrading go version to 1.17

@randall77
Copy link
Contributor

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

@seankhliao seankhliao changed the title panic: math/big: mismatched montgomery number lengths math/big: panic: mismatched montgomery number lengths Sep 18, 2021
@seankhliao seankhliao added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Sep 18, 2021
@FiloSottile
Copy link
Contributor

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

@fisherFIAF
Copy link
Author

can I resolve this problem by upgrading go version to 1.17

Worth trying. Since we don't know what the problem is, it is hard to be sure.

thanks very much, we will have a try.

@fisherFIAF
Copy link
Author

The stack trace shows a panic during a signature operation with an RSA certificate key. That's good news and bad news: the good news is that it's less likely to be attacker-controlled, the bads news is that it might be very hard for you to provide a reproducer without disclosing your certificate private key, and I'm not sure we can make much progress from the stack trace alone.

I got it, thank you very much.

@randall77
Copy link
Contributor

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

@fisherFIAF
Copy link
Author

I've been running this program for 24h, no errors yet.

package main

import (
	"fmt"
	"math/big"
	"math/rand"
	"time"
	"unsafe"
)

func main() {
	n := big.NewInt(0)
	q := big.NewInt(0)
	r := big.NewInt(0)

	w := big.NewInt(1)
	w.Lsh(w, 1024)
	w2 := big.NewInt(1)
	w2.Lsh(w, 2048)

	rnd := rand.New(rand.NewSource(time.Now().UnixNano()))

	for {
		n.Rand(rnd, w)

		q.QuoRem(w2, n, r)

		p := (*Int)(unsafe.Pointer(r))
		if r.Sign() < 0 || r.Cmp(n) >= 0 || len(p.abs) > 16 {
			fmt.Printf("w2=%x\n", w2)
			fmt.Printf("n=%x\n", n)
			fmt.Printf("q=%x\n", q)
			fmt.Printf("r=%x\n", r)
			panic("bad")
		}
	}
}

type Int struct {
	neg bool // sign
	abs nat  // absolute value of the integer
}
type nat []Word
type Word uint

If there's something wrong with nat.div, it's hard to find randomly.
Not sure what would be different about the use here. Probably n=p*q with p and q random primes, but hard to see how that would have an effect on how div works.

we run it on the server, every GRPC request create a new connection, hunders connection very seconds, it panic randomly recently, but not often, and I don't know how it happened either. I think we should reduce the connection times first.

@seankhliao seankhliao added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 12, 2022
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@ericlagergren
Copy link
Contributor

ericlagergren commented Jul 13, 2022 •
edited

Just another data point (since this got bumped by gopherbot): I've been running Keith's program for over a cumulative 48 hours across 3 machines (both amd64 and arm64) and haven't seen any errors, either.

@golang golang locked and limited conversation to collaborators Jul 13, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@FiloSottile @fisherFIAF @ericlagergren @randall77 @gopherbot @seankhliao