Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

compress/flate: index out of bounds on >2GB input #3676

Closed
gopherbot opened this issue May 28, 2012 · 8 comments
Closed

compress/flate: index out of bounds on >2GB input #3676

gopherbot opened this issue May 28, 2012 · 8 comments

Comments

@gopherbot
Copy link

by painted.now:

What steps will reproduce the problem?

0. Download data file http://jumbofiles.com/m7jl16x3shgi (password: "golang")
1. Decompress file as in.txt
1. Run http://play.golang.org/p/DfL3qASZmw and compress file "in.txt|
2. Wait for ~ 1.000.000 lines to be processed

What is the expected output?

1. A compressed gzip file of the given input (gzip can do it...)

What do you see instead?

panic: runtime error: index out of range

goroutine 1 [running]:
compress/flate.(*compressor).findMatch(0xf84005b160, 0x7ffb7fff0000e46c, 0x1b9400000002,
0x2, 0x407900, ...)
    /home/m/software/go/src/pkg/compress/flate/deflate.go:157 +0x11a
compress/flate.(*compressor).deflate(0xf84005b160, 0xf84028c000)
    /home/m/software/go/src/pkg/compress/flate/deflate.go:266 +0x410
compress/flate.(*compressor).write(0xf84005b160, 0xf84028cae8, 0x15180000066e,
0xf800001156, 0x0, ...)
    /home/m/software/go/src/pkg/compress/flate/deflate.go:359 +0xa0
compress/flate.(*Writer).Write(0xf84005b160, 0xf84028c000, 0x200000001156, 0xf800001156,
0x0, ...)
    /home/m/software/go/src/pkg/compress/flate/deflate.go:469 +0x54
compress/gzip.(*Writer).Write(0xf84005b0b0, 0xf84028c000, 0x200000001156,
0x7f6ae4e5ae48, 0x100000001, ...)
    /home/m/software/go/src/pkg/compress/gzip/gzip.go:190 +0x56a
fmt.Fprintf(0xf84002e750, 0xf84005b0b0, 0x4a9dcc, 0xa732500000003, 0x7f6ae4e5ae48, ...)
    /home/m/software/go/src/pkg/fmt/print.go:214 +0xa4
main.main()
    /home/m/software/go/compress.go:60 +0x854

goroutine 2 [syscall]:
created by runtime.main
    /home/m/software/go/src/pkg/runtime/proc.c:221


Which compiler are you using (5g, 6g, 8g, gccgo)?

6g (x64_64)

Which operating system are you using?

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=11.10
DISTRIB_CODENAME=oneiric
DISTRIB_DESCRIPTION="Ubuntu 11.10"

Which version are you using?  (run 'go version')

go version weekly.2012-03-27

Please provide any additional information below.

I know that you probably would not process the data line by line, however, to my
understanding the program should not crash.

Attachments:

  1. compress.go (1489 bytes)
@rsc
Copy link
Contributor

rsc commented May 29, 2012

Comment 1:

With some debugging prints:
chainHead=0 hashOffset=-2147188735 diff=2147188735
BAD i=2147188735 length=2 len(win)=58734
panic: runtime error: index out of range
goroutine 1 [running]:
compress/flate.(*compressor).findMatch(0x42191160, 0x7ffb7fff0000e46c,
0x1b9400000002, 0x2, 0x42179100, ...)
    /Users/rsc/g/go/src/pkg/compress/flate/deflate.go:160 +0x29c
compress/flate.(*compressor).deflate(0x42191160, 0x422bc400)
    /Users/rsc/g/go/src/pkg/compress/flate/deflate.go:270 +0x4ea
compress/flate.(*compressor).write(0x42191160, 0x422bcee8,
0x66e0000066e, 0x1156, 0x0, ...)
    /Users/rsc/g/go/src/pkg/compress/flate/deflate.go:363 +0x9f
compress/flate.(*Writer).Write(0x42191160, 0x422bc400, 0x115600001156,
0x100000001156, 0x0, ...)
    /Users/rsc/g/go/src/pkg/compress/flate/deflate.go:473 +0x54
compress/gzip.(*Writer).Write(0x421910b0, 0x422bc400, 0x115600001156,
0x115600001156, 0x0, ...)
    /Users/rsc/g/go/src/pkg/compress/gzip/gzip.go:190 +0x56c
main.main()
    /Users/rsc/Downloads/x.go:64 +0x9f8

@rsc
Copy link
Contributor

rsc commented May 29, 2012

Comment 2:

Thanks for the reproduction case. If I bound the arguments to findMatch the problem goes
away (diff below) but I am concerned that there are other wraparound bugs lurking.  I
will ask the person who wrote the code initially.
diff -r 7637a893613f src/pkg/compress/flate/deflate.go
--- a/src/pkg/compress/flate/deflate.go Tue May 29 14:37:41 2012 -0400
+++ b/src/pkg/compress/flate/deflate.go Tue May 29 15:43:15 2012 -0400
@@ -154,6 +154,9 @@
    minIndex := pos - windowSize
 
    for i := prevHead; tries > 0; tries-- {
+if i < 0 || i+1 >= len(win) || i+length >= len(win) || i+length < 0 {
+   print("BAD i=", i, " length=", length, " len(win)=", len(win), "\n")
+}
        if w0 == win[i] && w1 == win[i+1] && wEnd == win[i+length] {
            // The hash function ensures that if win[i] and win[i+1] match, win[i+2] matches
 
@@ -260,10 +263,11 @@
            minIndex = 0
        }
 
-       if d.chainHead-d.hashOffset >= minIndex &&
+       if off := d.chainHead-d.hashOffset; off >= minIndex && off < len(d.window) &&
            (d.fastSkipHashing != skipNever && lookahead > minMatchLength-1 ||
                d.fastSkipHashing == skipNever && lookahead > prevLength && prevLength < d.lazy) {
-           if newLength, newOffset, ok := d.findMatch(d.index, d.chainHead-d.hashOffset,
minMatchLength-1, lookahead); ok {
+if d.chainHead-d.hashOffset > 1<<29 { print("chainHead=", d.chainHead, "
hashOffset=", d.hashOffset, " diff=", off, "\n") }
+           if newLength, newOffset, ok := d.findMatch(d.index, off, minMatchLength-1,
lookahead); ok {
                d.length = newLength
                d.offset = newOffset
            }

Labels changed: added priority-later, removed priority-triage.

Status changed to Accepted.

@gopherbot
Copy link
Author

Comment 3 by painted.now:

Just wanted to add that the problem persists in go version "go1.0.1", I only reported
the problem for go version "weekly.2012-03-27", sorry.

@rsc
Copy link
Contributor

rsc commented May 29, 2012

Comment 4:

No problem; I have reproduced this with the current tip.

@krasin
Copy link

krasin commented May 30, 2012

Comment 5:

Reproduced. Started to work on this.

@krasin
Copy link

krasin commented May 30, 2012

Comment 6:

The fix is on the review: http://golang.org/cl/6249067/
It's completely my fault.

@krasin
Copy link

krasin commented May 31, 2012

Comment 7:

The fix is committed. Russ, should the bug be marked as closed?

@rsc
Copy link
Contributor

rsc commented May 31, 2012

Comment 8:

Yes, sorry. Usually we write "Fixes issue #3676." in the CL description and then it
closes automatically. I missed that that was missing.

Status changed to Fixed.

@golang golang locked and limited conversation to collaborators Jun 24, 2016
This issue was closed.
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

3 participants