Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

math/rand: index out of range in (*rngSource).Int63 #21099

Closed
erikdubbelboer opened this issue Jul 20, 2017 · 7 comments
Closed

math/rand: index out of range in (*rngSource).Int63 #21099

erikdubbelboer opened this issue Jul 20, 2017 · 7 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@erikdubbelboer
Copy link
Contributor

I'm not sure if reporting this is useful as I can't replicate this at all. Maybe it was just a cosmic ray or maybe someone can use the stack trace to find some obscure bug.

What version of Go are you using (go version)?

go version go1.8.3 linux/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/erik"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build313906869=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

Run a program that receives around 5000 requests per second using https://github.com/valyala/fasthttp

What did you expect to see?

No crashes

What did you see instead?

A very unusual panic in the Go source that I haven't seen before:

panic: runtime error: index out of range

goroutine 471249678 [running]:
math/rand.(*rngSource).Int63(0xc427177500, 0x0)
        /usr/local/go/src/math/rand/rng.go:231 +0x8c
math/rand.(*Rand).Int63(0xc4260b7fb0, 0x0)
        /usr/local/go/src/math/rand/rand.go:81 +0x33
math/rand.(*Rand).Int63-fm(0x0)
        /usr/local/go/src/math/rand/rand.go:211 +0x2a
math/rand.read(0xc46fdd8500, 0x100, 0x100, 0xc5a8135088, 0xc4260b7fd0, 0xc4260b7fd8, 0xc46fdd8500, 0x100, 0xc5a8135120)
        /usr/local/go/src/math/rand/rand.go:219 +0x9a
math/rand.(*Rand).Read(0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x20, 0xc46c04bde0, 0x0)
        /usr/local/go/src/math/rand/rand.go:211 +0x110
io.ReadAtLeast(0xf83d40, 0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x100, 0xb76620, 0x1, 0xc46c04bde0)
        /usr/local/go/src/io/io.go:307 +0xa9
io.ReadFull(0xf83d40, 0xc4260b7fb0, 0xc46fdd8500, 0x100, 0x100, 0x100, 0xc8b1fd730f65f11f, 0xd6990624d192e819)
        /usr/local/go/src/io/io.go:325 +0x58
crypto/rand.Int(0xf83d40, 0xc4260b7fb0, 0xc42773fb60, 0x410f02, 0xc468f70ea0, 0x120)
        /usr/local/go/src/crypto/rand/util.go:122 +0x14a
crypto/rsa.decrypt(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0xc5a8135430, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/rsa/rsa.go:499 +0x5f7
crypto/rsa.decryptAndCheck(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0xc5a8135430, 0xc5a8135430, 0x100, 0x0)
        /usr/local/go/src/crypto/rsa/rsa.go:559 +0x56
crypto/rsa.SignPKCS1v15(0xf83d40, 0xc4260b7fb0, 0xc4276bf320, 0x5, 0xc46c04bdc0, 0x20, 0x20, 0x40ef28, 0xb62060, 0xc4852c0c58, ...)
        /usr/local/go/src/crypto/rsa/pkcs1v15.go:250 +0x2bb
crypto/rsa.(*PrivateKey).Sign(0xc4276bf320, 0xf83d40, 0xc4260b7fb0, 0xc46c04bdc0, 0x20, 0x20, 0xf84bc0, 0xc4852c0c58, 0x0, 0x0, ...)
        /usr/local/go/src/crypto/rsa/rsa.go:105 +0x167
crypto/tls.(*ecdheKeyAgreement).generateServerKeyExchange(0xc42cd72e10, 0xc425fa4300, 0xc4276bf440, 0xc4302eb040, 0xc47be936c0, 0x10d8, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/key_agreement.go:276 +0x565
crypto/tls.(*serverHandshakeState).doFullHandshake(0xc5a8135a58, 0xc5a8135a00, 0x0)
        /usr/local/go/src/crypto/tls/handshake_server.go:404 +0x3f3
crypto/tls.(*Conn).serverHandshake(0xc4f9914e00, 0xc4aa08, 0xc4f9914f20)
        /usr/local/go/src/crypto/tls/handshake_server.go:85 +0x325
crypto/tls.(*Conn).Handshake(0xc4f9914e00, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1309 +0x2db
crypto/tls.(*Conn).Read(0xc4f9914e00, 0xc55b8d22c8, 0x1, 0x1, 0x0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:1117 +0x59
github.com/erikdubbelboer/fasthttp.acquireByteReader(0xc5a8135e48, 0xf908c0, 0xc4f9914e00, 0xc438725b00)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1807 +0x146
github.com/erikdubbelboer/fasthttp.(*Server).serveConn(0xc4277c9cc0, 0xf908c0, 0xc4f9914e00, 0xc42739c901, 0xc427390101)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1497 +0x156d
github.com/erikdubbelboer/fasthttp.(*Server).(github.com/erikdubbelboer/fasthttp.serveConn)-fm(0xf908c0, 0xc4f9914e00, 0xc5a8135f78, 0x1)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/server.go:1273 +0x3e
github.com/erikdubbelboer/fasthttp.(*workerPool).workerFunc(0xc42739c980, 0xc437cb3160)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:210 +0xde
github.com/erikdubbelboer/fasthttp.(*workerPool).getCh.func1(0xc42739c980, 0xc437cb3160, 0xaf1f40, 0xc437cb3160)
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:182 +0x35
created by github.com/erikdubbelboer/fasthttp.(*workerPool).getCh
        /home/erik/src/github.com/erikdubbelboer/fasthttp/workerpool.go:184 +0x135
@ALTree ALTree changed the title math: index out of range in math/rand.(*rngSource).Int63 math/rand: index out of range in (*rngSource).Int63 Jul 20, 2017
@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 20, 2017
@ALTree
Copy link
Member

ALTree commented Jul 20, 2017 •
edited

The out-of-bounds must be on rng.vec in rng.go (the stack trace says line 231 because Uint64() is inlined there).

rng.vec is only accessed using rng.tap and rng.feed, which are only changed by decrementing them, and when they become negative we add _LEN (which is the size of rng.vec).

rng.tap--
if rng.tap < 0 {
    rng.tap += _LEN
}

So when we use them to access the vector, both rng.tap and rng.feed can't be negative and can't be bigger than _LEN - 1. I don't see where the out-of-bounds panic could come from.

Considering you can't reproduce this, it smells like memory corruption (possibly caused by a data race).

@erikdubbelboer
Copy link
Contributor Author

I didn't look at the code yet but I agree that it shouldn't be possible to have an out-of-bounds panic here.

I don't think it's in a data race in our process as that would normally make it happen much more of then than this once in a year.

This was on a Google Compute instance which wasn't doing anything else. Not sure how susceptible these are to memory corruptions from something else but I guess it can happen.

@ALTree
Copy link
Member

ALTree commented Jul 20, 2017 •
edited

It's also possible that the memory corruption was caused by a bug in the Go runtime.

This issue, thought, as it is, it's not actionable: the stack trace tells us nothing, and you can't reproduce the crash, so we cannot use your report to extract a reproducer for a (possible) runtime bug.

Leaving this open in case someone else want to say something about this.

@bradfitz
Copy link
Contributor

fasthttp plays games with unsafe and sharing memory between []byte and string so my appetite for debugging this is minimal.

@bradfitz bradfitz added this to the Unplanned milestone Jul 20, 2017
@erikdubbelboer
Copy link
Contributor Author

I'm not sure the unsafe behavior of fasthttp is to blame, I would expect it to happen much more often in this case. I think it was just some random memory corruption so it will be impossible to debug.

@erikdubbelboer
Copy link
Contributor Author

I found my mistake. As you can see from the stack trace I am using tls but I supplied it with math/rand.Source which is not thread safe. tls requires a safe random source.

@ALTree
Copy link
Member

ALTree commented Jul 26, 2017

@erikdubbelboer thanks for the update!

@AkihiroSuda AkihiroSuda mentioned this issue Feb 7, 2018
Merged
@golang golang locked and limited conversation to collaborators Jul 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@bradfitz @erikdubbelboer @ALTree @gopherbot