Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/compile: large arrays cause large binaries and toolchain errors #17378

Open
crawshaw opened this issue Oct 7, 2016 · 9 comments
Open

cmd/compile: large arrays cause large binaries and toolchain errors #17378

crawshaw opened this issue Oct 7, 2016 · 9 comments
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Milestone

Comments

@crawshaw
Copy link
Member

crawshaw commented Oct 7, 2016

On tip, go version devel +0db9518 Fri Oct 7 03:46:50 2016 +0000 darwin/amd64:

$ cat j.go 
package main

var x = [1 << 30]byte{0: 1}

func main() {
    println(x[0])
}
$ go build j.go
$ ls -l j
-rwxr-xr-x  1 crawshaw  eng  1074669392 Oct  7 10:34 j

Bumping 1<<30 to 1<<31 turns it into a linker error:

~$ go build j.go 
# command-line-arguments
main.x: symbol too large (2147483648 bytes)
too much data in section SNOPTRDATA (over 2000000000 bytes)
too much data in section SDATA (over 2000000000 bytes)
too much data in section SBSS (over 2000000000 bytes)
too much data in section SNOPTRBSS (over 2000000000 bytes)
main.init: pc-relative relocation address for main.initdone· is too big: 0x800ad36a
main.init: pc-relative relocation address for main.initdone· is too big: 0x800ad355
runtime.alginit: pc-relative relocation address for runtime.cpuid_ecx is too big: 0x800ac585
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac72d
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac70b
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac700
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac701
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6f6
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6f7
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6ec
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6ed
runtime.alginit: pc-relative relocation address for runtime.hashkey is too big: 0x800ac6e2
runtime.alginit: pc-relative relocation address for runtime.useAeshash is too big: 0x800ac4ea
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a0b
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a0d
runtime.alginit: pc-relative relocation address for runtime.algarray is too big: 0x80091a1f
/Users/crawshaw/go/pkg/tool/darwin_amd64/link: too many errors

(Inspired by https://codegolf.stackexchange.com/questions/69189/build-a-compiler-bomb/69193#69193)

@mrjrieke
Copy link

mrjrieke commented Oct 7, 2016

I wonder how compiler behaves if allocate is inside main?

@ALTree
Copy link
Member

ALTree commented Oct 7, 2016

Isn't this on purpose? The message was introduced in commit cmd/link: reject data size > 2 GB to fix issue #9862

@mrjrieke
Copy link

mrjrieke commented Oct 7, 2016

True Alberto. Although, does golint help identify the line number with the
problem? Seems like something either the compiler or one of the tools
should be able to identify more precisely the source of the problem.

On Fri, Oct 7, 2016 at 9:52 AM, Alberto Donizetti notifications@github.com
wrote:

Isn't this on purpose? The message was introduced in commit cmd/link:
reject data size > 2 GB to fix issue #9862
#9862


You are receiving this because you commented.
Reply to this email directly, view it on GitHub
#17378 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/ATlFP4yqQAQDLDsBfd-FPyaigmcc4QzUks5qxnjrgaJpZM4KRGlQ
.

@quentinmit quentinmit added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Oct 9, 2016
@quentinmit quentinmit added this to the Go1.8 milestone Oct 9, 2016
@rsc rsc modified the milestones: Go1.9Early, Go1.8 Oct 21, 2016
@bradfitz bradfitz modified the milestones: Go1.9Maybe, Go1.9Early May 3, 2017
@bradfitz bradfitz modified the milestones: Go1.9Maybe, Go1.10 Jul 20, 2017
@mdempsky mdempsky modified the milestones: Go1.10, Go1.11 Nov 30, 2017
@mdempsky
Copy link
Member

It's not obvious to me that this issue is actionable.

@mariecurried
Copy link

mariecurried commented Apr 6, 2018

I encountered this issue too.
The simplest case I could come up with was https://play.golang.org/p/EK8zDu5vVQN (creates a ~800MB executable).
The thing I find weird is that https://play.golang.org/p/LM9TBaQcRIo works just fine and their assembly code isn't that different (when compiling with 'go tool compile -S ...', because after using objdump the code is totally different).

@mariecurried
Copy link

mariecurried commented Apr 7, 2018

I think I found the value where the first jump in the size of the executable occurs:

  • When I use an array size of 1310720, the executable is ~1MB;
  • When I change it to 1310721, the executable is ~11MB.

Hope this helps debug the problem.

@gopherbot gopherbot modified the milestones: Go1.11, Unplanned May 23, 2018
@gopherbot gopherbot added the compiler/runtime Issues related to the Go compiler and/or runtime. label Jul 13, 2022
@ericjing83
Copy link

Hi, is there any updates?
I prefer to use large global array to store my web service cache because large global array is not allocated on heap, it will be allocated on global storage zone, just like C++. So, large global array will have higher performance than large global slice which is allocated on heap.

@ericjing83
Copy link

strangely, when I do below 2 things, above issue seems disappear
1.update the golang version to 1.20
2. update /etc/security/limits.conf of my debian server:
root soft memlock unlimited
root hard memlock unlimited
root soft nofile 3100000
root hard nofile 3100000
root soft stack 67108864
root hard stack 67108864
ericjing soft memlock unlimited
ericjing hard memlock unlimited
ericjing soft nofile 3100000
ericjing hard nofile 3100000
ericjing soft stack 67108864
ericjing hard stack 67108864

my test code are as follow, from global zone, I defined a struct and an array like this, when I compile the golang project, the execuable is small (less than 2MB):
type TaskObject struct {
UserID int
RequestData []byte
}
var Shard [4096][4096]TaskObject //Shard is allocated in global storage zone, just like c++, I have done heap escape analysis by "go build -gcflags=-m main.go", and I'm sure this global variable is NOT allocated on heap. Even if I increase the size to 8192, the execuable is still small.

@ericjing83
Copy link

And, when I use a one big dimentional array (size = 4096*4096), the execuable is still small, and my golang program runs fine on my debian server.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
compiler/runtime Issues related to the Go compiler and/or runtime. NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made.
Projects
None yet
Development

No branches or pull requests

10 participants