Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

x/tools/go/ssa: requires tons of memory for testing #14113

Closed
mikioh opened this issue Jan 27, 2016 · 10 comments
Closed

x/tools/go/ssa: requires tons of memory for testing #14113

mikioh opened this issue Jan 27, 2016 · 10 comments
Assignees
@alandonovan
Labels
Tools This label describes issues relating to any tools in the x/tools repository.
Milestone
Unreleased

Comments

@mikioh
Copy link
Contributor

mikioh commented Jan 27, 2016

For example, on linux-amd64-nocgo (http://build.golang.org/log/00d8b3a0c6186837e4e3d83cc06a69797734f202),

ok      golang.org/x/tools/go/pointer   57.885s
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0x85dd00, 0x16)
    /tmp/workdir/go/src/runtime/panic.go:527 +0x90
runtime.sysMap(0xc8628c0000, 0x100000, 0xbd100, 0xa14418)
    /tmp/workdir/go/src/runtime/mem_linux.go:203 +0x9b
runtime.mHeap_SysAlloc(0x9f4940, 0x100000, 0x0)
    /tmp/workdir/go/src/runtime/malloc.go:426 +0x160
runtime.mHeap_Grow(0x9f4940, 0x8, 0x0)
    /tmp/workdir/go/src/runtime/mheap.go:628 +0x63
runtime.mHeap_AllocSpanLocked(0x9f4940, 0x1, 0x7f0aeb59d390)
    /tmp/workdir/go/src/runtime/mheap.go:532 +0x5f1
runtime.mHeap_Alloc_m(0x9f4940, 0x1, 0x9, 0x7f0aeb59d390)
    /tmp/workdir/go/src/runtime/mheap.go:425 +0x1ac
runtime.mHeap_Alloc.func1()
    /tmp/workdir/go/src/runtime/mheap.go:484 +0x41
runtime.systemstack(0xc820167e50)
    /tmp/workdir/go/src/runtime/asm_amd64.s:278 +0xab
runtime.mHeap_Alloc(0x9f4940, 0x1, 0x10000000009, 0x40db44)
    /tmp/workdir/go/src/runtime/mheap.go:485 +0x63
runtime.mCentral_Grow(0x9fc778, 0x0)
    /tmp/workdir/go/src/runtime/mcentral.go:190 +0x93
runtime.mCentral_CacheSpan(0x9fc778, 0x7f0aeb59d390)
    /tmp/workdir/go/src/runtime/mcentral.go:86 +0x4d4
runtime.mCache_Refill(0x7f0aec5a3000, 0x9, 0x7f0aeb59d390)
    /tmp/workdir/go/src/runtime/mcache.go:118 +0xcf
runtime.mallocgc.func2()
    /tmp/workdir/go/src/runtime/malloc.go:614 +0x2b
runtime.systemstack(0xc820016000)
    /tmp/workdir/go/src/runtime/asm_amd64.s:262 +0x79
runtime.mstart()
    /tmp/workdir/go/src/runtime/proc1.go:668

(snip)

created by golang.org/x/tools/go/ssa.(*Program).Build
    /tmp/workdir/gopath/src/golang.org/x/tools/go/ssa/builder.go:2248 +0x12a
FAIL    golang.org/x/tools/go/ssa   28.128s
@mikioh mikioh added this to the Unreleased milestone Jan 27, 2016
@bradfitz
Copy link
Contributor

/cc @alandonovan @griesemer

As I told Robert, these machines have 1.8GB of memory to do nothing but run all.bash, essentially.

@griesemer
Copy link
Contributor

The question though is: Do these machines have less memory than the others?
Because these tests run fine on the other platforms.

  • gri

On Tue, Jan 26, 2016 at 11:29 PM, Brad Fitzpatrick <notifications@github.com

wrote:

/cc @alandonovan https://github.com/alandonovan @griesemer
https://github.com/griesemer

As I told Robert, these machines have 1.8GB of memory to do nothing but
run all.bash, essentially.

—
Reply to this email directly or view it on GitHub
#14113 (comment).

@bradfitz
Copy link
Contributor

Nope. Same. https://github.com/golang/build/blob/master/dashboard/builders.go

The race builders are bigger, but the rest are the same.

@bradfitz
Copy link
Contributor

I can reproduce this on Linux with no special care on my development machine (not a builder).

Our tests should not require multiple gigabytes of memory.

I'm going to disable it.

@alandonovan, @griesemer, if these tests are important, put them on a diet. I don't know what they're doing, but it's too much.

@bradfitz
Copy link
Contributor

I saw it running two tests at once, one using 1.2GB and one using 700+MB. That's the limit of the machine. If the Linux builders have no swap, that explains their death. It's possible other machines are just swapping a bunch.

@cespare
Copy link
Contributor

cespare commented Jan 28, 2016

The only heavyweight test is TestStdlib which "runs the SSA builder in sanity-checking mode on all packages beneath $GOROOT". That test uses 1.3GB to run on my Linux machine.

Perhaps that test should use a small subset of the stdlib packages by default.

Alternatively, the test could collect its statistics package-by-package rather than loading all packages at once, but it does do one global function uniqueness check that would be incompatible with this approach.

@griesemer
Copy link
Contributor

We should just have a -short version: In that case, the test runs just over
say $GOROOT/src/go.

  • gri

On Wed, Jan 27, 2016 at 4:06 PM, Caleb Spare notifications@github.com
wrote:

The only heavyweight test is TestStdlib which "runs the SSA builder in
sanity-checking mode on all packages beneath $GOROOT". That test uses 1.3GB
to run on my Linux machine.

Perhaps that test should use a small subset of the stdlib packages by
default.

Alternatively, the test could collect its statistics package-by-package
rather than loading all packages at once, but it does do one global
function uniqueness check that would be incompatible with this approach.

—
Reply to this email directly or view it on GitHub
#14113 (comment).

gopherbot pushed a commit to golang/tools that referenced this issue Jan 28, 2016
@bradfitz
all: skip slow tests in short mode
2fd075e 
Updates golang/go#14113
Updates golang/go#11811

Change-Id: I61851de12ff474d3b738fc88f402742677973cae
Reviewed-on: https://go-review.googlesource.com/18992
Reviewed-by: Robert Griesemer <gri@golang.org>
gopherbot pushed a commit to golang/tools that referenced this issue Jan 28, 2016
@bradfitz
all: skip slow tests in short mode
03f58a3 
Updates golang/go#14113
Updates golang/go#11811

Change-Id: I61851de12ff474d3b738fc88f402742677973cae
@alandonovan alandonovan self-assigned this Nov 28, 2017
@gopherbot gopherbot added the Tools This label describes issues relating to any tools in the x/tools repository. label Sep 12, 2019
@bcmills bcmills mentioned this issue May 21, 2020
Closed
@gopherbot
Copy link

Change https://golang.org/cl/360837 mentions this issue: go/pointer: skip TestInput on 32-bit platforms

gopherbot pushed a commit to golang/tools that referenced this issue Nov 2, 2021
go/pointer: skip TestInput on 32-bit platforms
182bbdc 
This test was already memory-hungry to begin with, and apparently the
switch to go/packages in CL 356513 pushed it over the edge of the
32-bit address space (at least with the default GOGC setting). Rather
than trying to precisely tune it to skim under the 32-bit limit,
let's just skip the test on platforms with insufficient address space.

Updates golang/go#14113
Updates golang/go#48547

Change-Id: Iab99e9ce70a98034194d7c7ad7df7a545ac95ef3
Reviewed-on: https://go-review.googlesource.com/c/tools/+/360837
Trust: Bryan C. Mills <bcmills@google.com>
Run-TryBot: Bryan C. Mills <bcmills@google.com>
gopls-CI: kokoro <noreply+kokoro@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Russ Cox <rsc@golang.org>
@gopherbot
Copy link

Change https://go.dev/cl/449518 mentions this issue: go/pointer: break TestInput into subtests and update skips

gopherbot pushed a commit to golang/tools that referenced this issue Nov 11, 2022
@gopherbot
go/pointer: break TestInput into subtests and update skips
e3b3c01 
TestInput is fiendishly slow and memory-hungry. Breaking it into
subtests reveals that the vast majority of that cost can be attributed
to a single test case: "testdata/a_test.go".

Update the address-space-based skip to skip only that one input, skip
it under the race detector (it is timing out on the new
"linux-amd64-longtest-race" builder), and run the remaining inputs in
parallel (to reduce test latency now that we've better identified the
culprit).

Updates golang/go#14113.
Updates golang/go#54630.

Change-Id: I105cfa173edc74692eaa41efd50ae08eeacf0d7d
Reviewed-on: https://go-review.googlesource.com/c/tools/+/449518
TryBot-Result: Gopher Robot <gobot@golang.org>
gopls-CI: kokoro <noreply+kokoro@google.com>
Auto-Submit: Bryan Mills <bcmills@google.com>
Run-TryBot: Bryan Mills <bcmills@google.com>
Reviewed-by: Alan Donovan <adonovan@google.com>
@bcmills bcmills mentioned this issue Apr 17, 2023
Open
@adonovan
Copy link
Member

The pointer analysis was deleted and go/ssa skips the big stdlib test in -short mode, so this is done.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@alandonovan alandonovan

Labels
Tools This label describes issues relating to any tools in the x/tools repository.
Projects
None yet
Milestone
Unreleased
Development

No branches or pull requests
7 participants
@bradfitz @mikioh @cespare @adonovan @alandonovan @griesemer @gopherbot