Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

index/suffixarray: oom in Read #52813

Closed
catenacyber opened this issue May 10, 2022 · 4 comments
Closed

index/suffixarray: oom in Read #52813

catenacyber opened this issue May 10, 2022 · 4 comments
Labels
FrozenDueToAge

Comments

@catenacyber
Copy link
Contributor

What version of Go are you using (go version)?

$ go version
go version go1.17.6 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/catena/Library/Caches/go-build"
GOENV="/Users/catena/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/catena/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/catena/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GOVCS=""
GOVERSION="go1.17.6"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/catena/go/src/github.com/catenacyber/go/src/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch x86_64 -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pp/dc1dtf9x2js3v0jx_m010nqr0000gn/T/go-build4237848497=/tmp/go-build -gno-record-gcc-switches -fno-common"
GOROOT/bin/go version: go version go1.17.6 darwin/amd64
GOROOT/bin/go tool compile -V: compile version go1.17.6
uname -v: Darwin Kernel Version 21.3.0: Wed Jan  5 21:37:58 PST 2022; root:xnu-8019.80.24~20/RELEASE_X86_64
ProductName:	macOS
ProductVersion:	12.2.1
BuildVersion:	21D62
lldb --version: lldb-1316.0.9.41
Apple Swift version 5.6 (swiftlang-5.6.0.323.62 clang-1316.0.20.8)
gdb --version: GNU gdb (GDB) 9.1

What did you do?

Run https://go.dev/play/p/ArMJGdfERMY

What did you expect to see?

The program finishing and printing Hello

What did you see instead?

Nothing (because of oom)

Profile shows up

index/suffixarray.(*Index).Read
/usr/local/go/src/index/suffixarray/suffixarray.go

  Total:     40.50GB    40.50GB (flat, cum)   100%
    168            .          .            
    169            .          .           	// allocate space 
    170            .          .           	if 2*n < cap(x.data) || cap(x.data) < n || x.sa.int32 != nil && n > maxData32 || x.sa.int64 != nil && n <= maxData32 { 
    171            .          .           		// new data is significantly smaller or larger than 
    172            .          .           		// existing buffers - allocate new ones 
    173       4.50GB     4.50GB           		x.data = make([]byte, n) 
    174            .          .           		x.sa.int32 = nil 
    175            .          .           		x.sa.int64 = nil 
    176            .          .           		if n <= maxData32 { 
    177            .          .           			x.sa.int32 = make([]int32, n) 
    178            .          .           		} else { 
    179         36GB       36GB           			x.sa.int64 = make([]int64, n) 
    180            .          .           		} 
    181            .          .           	} else { 
    182            .          .           		// re-use existing buffers 
    183            .          .           		x.data = x.data[0:n] 
    184            .          .           		x.sa = x.sa.slice(0, n)

Found by https://github.com/catenacyber/ngolo-fuzzing on oss-fuzz
https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=46673

Is this a bug ?
@ZekeLu
Copy link
Contributor

ZekeLu commented May 10, 2022 •
edited

This one is similar to #52120. I don't think this is a bug either.

The invalid data passed to Read() indicates there are 4831828922 items, so Read() will make slices to hold those items, which results in the oom.

import (
	"encoding/binary"
	"fmt"
)

func ExampleVarint() {
	// the first 10 bytes holds the number of the items in the suffix array
	x, _ := binary.Varint([]byte{0xf4, 0xee, 0xfe, 0xff, 0x23, 0x0, 0x0, 0xb, 0x2d, 0x10})
	fmt.Println(x)
	// Output:
	// 4831828922
}

@catenacyber
Copy link
Contributor Author

Here is an example of a oom fix : https://go-review.googlesource.com/c/go/+/400378/

But I am okay if it is not considered a bug

@randall77
Copy link
Contributor

You are right that we have varying degrees of "robust with respect to bad input" in the stdlib.

That CL was for a package that reads external files, which may come from any source (not just Go). Conceivably an adversary might invent a PE file and convince your Go code to try and read it.

index/suffixarray can only read things that index/suffixarray wrote. I don't think anyone is accepting untrusted index/suffixarray -encoded data and trying to read it in. (I could be wrong though.)

@catenacyber
Copy link
Contributor Author

👍 Thanks for the details

@golang golang locked and limited conversation to collaborators May 11, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@ZekeLu @randall77 @gopherbot @catenacyber