Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: test deals poorly with running out of memory #47467

Closed
rlpowell opened this issue Jul 30, 2021 · 4 comments
Closed

cmd/go: test deals poorly with running out of memory #47467

rlpowell opened this issue Jul 30, 2021 · 4 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@rlpowell
Copy link

What version of Go are you using (go version)?

root@54a4d2c88906:/go# go version
go version go1.16.6 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
root@54a4d2c88906:/go# go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.16.6"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build1878842124=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ docker run -v $(pwd):/go/src --rm -it golang:1.16.6 bash -c 'cd src ; go test'
PASS
ok      example.com/user/hello  3.343s
$ docker run -v $(pwd):/go/src --rm -m 100M -it golang:1.16.6 bash -c 'cd src ; go test'
signal: killed
FAIL    example.com/user/hello  2.025s


$ cat main_test.go
package main

import "testing"

func TestReverseRunes(t *testing.T) {
  bytes := make([]byte,1)
  for i := 0; i < (1024*1024*1024); i++ {
    bytes = append(bytes,1)
  }
}

What did you expect to see?

Some kind of error that indicates which test failed or why it failed or something; doing this same thing with a bunch of tests behaves the same way, it just stops. It would be really, really nice if it said "I ran out of memory", but I don't know how possible that is.

What did you see instead?

"signal: killed"
@rlpowell rlpowell mentioned this issue Jul 30, 2021
Closed
@ALTree ALTree changed the title "go test" deals poorly with running out of memory cmd/go: test deals poorly with running out of memory Jul 30, 2021
@ALTree
Copy link
Member

ALTree commented Jul 30, 2021 •
edited

It is indeed not obvious what that failure message means. Here's a concrete example where the gob test suite was failing due to a OOM, and it took a while to realize that an OOM was the root cause: #31517.

On the other hand, I don't know if it's even possible to catch that specific panic reason and print a special report for it.

cc @aclements @mknyszek

@ALTree ALTree added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Jul 30, 2021
@ALTree ALTree added this to the Unplanned milestone Jul 30, 2021
@jayconrod
Copy link
Contributor

go test could probably be a bit more explicit when a test process for some specific package terminates due to a signal.

That said, I don't know that it's possible to distinguish a child process being terminated by the OOM killer from a process terminated by something else that sent SIGKILL.

@heschi
Copy link
Contributor

heschi commented Jul 30, 2021

To the best of my knowledge, on Linux the only ways to find out about an OOM kill are to scrape dmesg or to have the process in a memcg and watch memory.oom_control.

@ianlancetaylor
Copy link
Contributor

I don't think we can reasonably do anything other than what we already do. To avoid the OOM killer on Linux you have to adjust /proc/sys/vm/overcommit_memory, which will have other consequences.

I'm going to close this issue because I don't think there is any reasonable action for us to take here. Sorry.

@golang golang locked and limited conversation to collaborators Jul 30, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
6 participants
@rlpowell @jayconrod @ianlancetaylor @ALTree @gopherbot @heschi