Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/link: Oversize object file (example uses generics) causes confusing "slice bounds out of range" panic #66357

Open
djedward opened this issue Mar 16, 2024 · 2 comments
Open

cmd/link: Oversize object file (example uses generics) causes confusing "slice bounds out of range" panic #66357

djedward opened this issue Mar 16, 2024 · 2 comments
Assignees
@cherrymui
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Backlog

Comments

@djedward
Copy link

Go version

go version go1.21.8 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/home/dj/.cache/go-build'
GOENV='/home/dj/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/home/dj/go/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/home/dj/go'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/go'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='auto'
GOTOOLDIR='/usr/local/go/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.21.8'
GCCGO='gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='g++'
CGO_ENABLED='1'
GOMOD='/home/dj/gosize/code/go.mod'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build2460092799=/tmp/go-build -gno-record-gcc-switches'

What did you do?

We updated our AWS SDK v1 version and existing code started failing. I distilled it down to this minimal case. The issue happens when the generated object in the GOCACHE goes above 4GB. The goobj/objfile Header appears to be using a uint32, which seems like an overflow might be happening when calculating the offsets. I dumped the headers from one of the objects while investigating (not the minimal case above, so offsets are a little different):

BlkAutolib: 3225526148
BlkPkgIdx: 3225526596
BlkFile: 3225526884
BlkSymdef: 3225528524
BlkHashed64def: 3225727751
BlkHasheddef: 3225729032
BlkNonpkgdef: 3226325537
BlkNonpkgref: 3226479026
BlkRefFlags: 3226587890
BlkHash64: 3226587890
BlkHash: 3226588378
BlkRelocIdx: 3227042858
BlkAuxIdx: 3227223910
BlkDataIdx: 3227404962
BlkReloc: 3227586014
BlkAux: 3228926500
BlkData: 3229305472
BlkRefName: 10123514
BlkEnd: 10148874

I tested it in a few other versions:
1.22.1 - compiles
1.22.0 - compiles
1.22rc2 - compiles
1.22rc1 - fails for same reason

What did you see happen?

panic: runtime error: slice bounds out of range [::3237855945] with capacity 26478551

goroutine 1 [running]:
cmd/internal/goobj.(*Reader).BytesAt(...)
	cmd/internal/goobj/objfile.go:622
cmd/internal/goobj.(*Reader).uint32At(...)
	cmd/internal/goobj/objfile.go:635
cmd/internal/goobj.(*Reader).StringRef(0x0?, 0x1be438?)
	cmd/internal/goobj/objfile.go:675 +0xf8
cmd/internal/goobj.(*Reader).Autolib(0xc000014120)
	cmd/internal/goobj/objfile.go:688 +0x71
cmd/link/internal/loader.(*Loader).Preload(0xc0000ec000, 0xa, 0xc000028020, 0xc0001c2000, 0xc0000cc410, 0x46f725?)
	cmd/link/internal/loader/loader.go:2147 +0x395
cmd/link/internal/ld.ldobj(0xc00019e000, 0xc000028020, 0xc0001c2000, 0x194086c, {0xc00001a300, 0x2c}, {0x7ffcd10911bc, 0x24})
	cmd/link/internal/ld/lib.go:2294 +0xa8f
cmd/link/internal/ld.loadobjfile(0xc00019e000, 0xc0001c2000)
	cmd/link/internal/ld/lib.go:1136 +0x7f3
cmd/link/internal/ld.(*Link).loadlib(0xc00019e000)
	cmd/link/internal/ld/lib.go:544 +0x214
cmd/link/internal/ld.Main(_, {0x20, 0x20, 0x1, 0x7, 0x10, 0x0, {0xc000012439, 0x1, 0x1}, ...})
	cmd/link/internal/ld/main.go:284 +0x104b
main.main()
	cmd/link/main.go:72 +0xdfb

What did you expect to see?

A successful build
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Mar 16, 2024
@cherrymui
Copy link
Member

Currently there is no plan to support an object larger than 4 GiB. Try refactoring the code or splitting to multiple objects. We can emit a better error, though.

@djedward
Copy link
Author

Sorry for not being clear, I'm not asking for >4GB support. Having a better error message would suffice. Especially because we are unblocked by just using Go 1.22 which generates a much smaller object. I didn't analyze the objects from 1.22, but the largest object in the cache is ~75MB for the same sample program.

I mainly opened the issue because the behavior is so vastly different between 1.21 and 1.22. The suspected overflow seems to result in strange offsets. The end offset in the object says it ends at about 10MB, but the previous offsets are in the 3GB range. I couldn't find another object in the cache that seems to have an end offset before the other offsets. So my thought was that if the overflow issue was addressed, it might actually produce a much smaller object file and be able to compile. But this is my first time looking at any of this code (I freely admit being a complete noob 😄), so it might be completely acceptable that those offsets are after the end offset.

@dr2chase dr2chase changed the title cmd/link: Using generics causes "slice bounds out of range" panic cmd/link: Oversize object file (example uses generics) causes confusing "slice bounds out of range" panic Mar 18, 2024
@dr2chase dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 18, 2024
@mknyszek mknyszek added BadErrorMessage Issues related compiler error messages that should be better. and removed BadErrorMessage Issues related compiler error messages that should be better. labels Mar 20, 2024
@mknyszek mknyszek added this to the Backlog milestone Mar 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@cherrymui cherrymui

Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
Go Compiler / Runtime
Status: Todo
Milestone
Backlog
Development

No branches or pull requests
5 participants
@mknyszek @dr2chase @djedward @gopherbot @cherrymui