Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

internal/fuzz,cmd/go: fuzz coordinator quickly runs out of address space on linux/386 #65434

Open
bcmills opened this issue Feb 1, 2024 · 2 comments
Open

internal/fuzz,cmd/go: fuzz coordinator quickly runs out of address space on linux/386 #65434

bcmills opened this issue Feb 1, 2024 · 2 comments
Labels
fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Milestone
Backlog

Comments

@bcmills
Copy link
Contributor

bcmills commented Feb 1, 2024

Go version

go version go1.22rc2 linux/amd64

Output of go env in your module/workspace:

GO111MODULE=''
GOARCH='amd64'
GOBIN=''
GOCACHE='/usr/local/google/home/bcmills/.cache/go-build'
GOENV='/usr/local/google/home/bcmills/.config/go/env'
GOEXE=''
GOEXPERIMENT=''
GOFLAGS=''
GOHOSTARCH='amd64'
GOHOSTOS='linux'
GOINSECURE=''
GOMODCACHE='/usr/local/google/home/bcmills/pkg/mod'
GONOPROXY=''
GONOSUMDB=''
GOOS='linux'
GOPATH='/usr/local/google/home/bcmills'
GOPRIVATE=''
GOPROXY='https://proxy.golang.org,direct'
GOROOT='/usr/local/google/home/bcmills/sdk/go1.22rc2'
GOSUMDB='sum.golang.org'
GOTMPDIR=''
GOTOOLCHAIN='local'
GOTOOLDIR='/usr/local/google/home/bcmills/sdk/go1.22rc2/pkg/tool/linux_amd64'
GOVCS=''
GOVERSION='go1.22rc2'
GCCGO='/usr/bin/gccgo'
GOAMD64='v1'
AR='ar'
CC='gcc'
CXX='c++'
CGO_ENABLED='1'
GOMOD='/dev/null'
GOWORK=''
CGO_CFLAGS='-O2 -g'
CGO_CPPFLAGS=''
CGO_CXXFLAGS='-O2 -g'
CGO_FFLAGS='-O2 -g'
CGO_LDFLAGS='-O2 -g'
PKG_CONFIG='pkg-config'
GOGCCFLAGS='-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -ffile-prefix-map=/tmp/go-build4207537818=/tmp/go-build -gno-record-gcc-switches'

What did you do?

GOARCH=386 go1.22rc2 test cmd/go -run=TestScript/test_fuzz_fuzztime

What did you see happen?

On my workstation (a 24 vCPU Xeon VM), the 32-bit test coordinator process crashes before its 5s fuzztime expires. It appears to run out of address space.

vcs-test.golang.org rerouted to http://127.0.0.1:40697
https://vcs-test.golang.org rerouted to https://127.0.0.1:37583
go test proxy running at GOPROXY=http://127.0.0.1:37953/mod
--- FAIL: TestScript (0.03s)
    --- FAIL: TestScript/test_fuzz_fuzztime (11.18s)
        script_test.go:132: 2024-02-01T18:21:37Z
        script_test.go:134: $WORK=/tmp/cmd-go-test-3219792428/tmpdir1553243795/test_fuzz_fuzztime2893856645
        script_test.go:156: 
            > [!fuzz] skip
            [condition not met]
            > [short] skip
            [condition not met]
            > env GOCACHE=$WORK/cache
            # There are no seed values, so 'go test' should finish quickly. (5.689s)
            # Fuzzing should exit 0 after fuzztime, even if timeout is short. (5.444s)
            > go test -timeout=3s -fuzz=FuzzFast -fuzztime=5s
            [stdout]
            warning: the test binary was not built with coverage instrumentation, so fuzzing will run without coverage guidance and may be inefficient
            warning: starting with empty corpus
            fuzz: elapsed: 0s, execs: 0 (0/sec)
            fuzz: elapsed: 3s, execs: 2296729 (765565/sec)
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            fatal error: out of memory
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            fatal error: out of memory
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            fatal error: out of memory
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            fatal error: out of memory
            runtime: out of memory: cannot allocate 104857600-byte block (1586888704 in use)
            fatal error: out of memory
            fatal error: out of memory
            
            goroutine 52 gp=0x9da9688 m=33 mp=0x9c81608 [running]:
            runtime.throw({0x81ff4fb, 0xd})
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/runtime/panic.go:1023 +0x4d fp=0xa4a772c sp=0xa4a7718 pc=0x808423d
            runtime.(*mcache).allocLarge(0xf7f2ee98, 0x63fff9c, 0x1)
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/runtime/mcache.go:236 +0x1c7 fp=0xa4a7758 sp=0xa4a772c pc=0x805d267
            runtime.mallocgc(0x63fff9c, 0x81d5b40, 0x1)
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/runtime/malloc.go:1165 +0x640 fp=0xa4a77c0 sp=0xa4a7758 pc=0x80553f0
            runtime.makeslice(0x81d5b40, 0x0, 0x63fff9c)
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/runtime/slice.go:107 +0x4f fp=0xa4a77d4 sp=0xa4a77c0 pc=0x809e0ef
            internal/fuzz.(*mutator).mutate(0xa6a6018, {0x9f8f030, 0x1, 0x1}, 0x6400000)
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/internal/fuzz/mutator.go:107 +0x4e3 fp=0xa4a7838 sp=0xa4a77d4 pc=0x81846f3
            internal/fuzz.(*workerClient).fuzz(0xa6a6030, {0x823e7ac, 0x9d90030}, {{0x0, 0x0}, {0x9d88010, 0x8}, {0x9d8c000, 0x1b, 0x20}, ...}, ...)
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/internal/fuzz/worker.go:1118 +0xc4f fp=0xa4a7a80 sp=0xa4a7838 pc=0x818e44f
            internal/fuzz.(*worker).coordinate(0x9d926e0, {0x823e7ac, 0x9d90030})
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/internal/fuzz/worker.go:156 +0x4dc fp=0xa4a7fb0 sp=0xa4a7a80 pc=0x81884fc
            internal/fuzz.CoordinateFuzzing.func3()
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/internal/fuzz/fuzz.go:185 +0x45 fp=0xa4a7ff0 sp=0xa4a7fb0 pc=0x81800b5
            runtime.goexit({})
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/runtime/asm_386.s:1363 +0x1 fp=0xa4a7ff4 sp=0xa4a7ff0 pc=0x80bd911
            created by internal/fuzz.CoordinateFuzzing in goroutine 9
            	/usr/local/google/home/bcmills/sdk/go1.22rc2/src/internal/fuzz/fuzz.go:184 +0x6d0
            
[…]
            exit status 2
            FAIL	fuzz	5.106s
        script_test.go:156: FAIL: testdata/script/test_fuzz_fuzztime.txt:9: go test -timeout=3s -fuzz=FuzzFast -fuzztime=5s: exit status 1
FAIL
FAIL	cmd/go	11.310s
FAIL

What did you expect to see?

All tests passing. Ideally, go test -fuzz should work reliably on 32-bit systems: it should avoid excessively large inputs, and should store existing inputs with an efficient summary/index and evict data out to temporary files on disk as needed to work within the program's available address space.
@bcmills bcmills added ToolSpeed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. fuzz Issues related to native fuzzing support labels Feb 1, 2024
@bcmills bcmills added this to the Backlog milestone Feb 1, 2024
@bcmills
Copy link
Contributor Author

bcmills commented Feb 1, 2024

(attn @golang/fuzzing)

@bcmills bcmills changed the title internal/fuzz,cmd/go: TestScript/test_fuzz_fuzztime fails on linux/386 on a reasonably fast machine internal/fuzz,cmd/go: fuzz coordinator quickly runs out of address space on linux/386 Feb 1, 2024
@gopherbot
Copy link

Change https://go.dev/cl/560515 mentions this issue: cmd/go: limit GOMAXPROCS in TestScript/test_fuzz_fuzztime

gopherbot pushed a commit that referenced this issue Feb 1, 2024
@gopherbot
cmd/go: limit GOMAXPROCS in TestScript/test_fuzz_.*
d864eac 
This limits the throughput and resource consumption of the fuzz
workers in the tests, which also reduces the likelihood of running out
of address space in the fuzz coordinator during the test.

(Ideally the coordinator should not be limited by address space;
this just works around the failure mode in the tests for now.)

For #65434.

Change-Id: I3086c6278d6803a3dbf17a46ed01b68cedc92ad9
Cq-Include-Trybots: luci.golang.try:gotip-linux-386-longtest,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/560515
Reviewed-by: Roland Shoemaker <roland@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
ezz-no pushed a commit to ezz-no/go-ezzno that referenced this issue Feb 18, 2024
@ezz-no
cmd/go: limit GOMAXPROCS in TestScript/test_fuzz_.*
f663b6b 
This limits the throughput and resource consumption of the fuzz
workers in the tests, which also reduces the likelihood of running out
of address space in the fuzz coordinator during the test.

(Ideally the coordinator should not be limited by address space;
this just works around the failure mode in the tests for now.)

For golang#65434.

Change-Id: I3086c6278d6803a3dbf17a46ed01b68cedc92ad9
Cq-Include-Trybots: luci.golang.try:gotip-linux-386-longtest,gotip-windows-amd64-longtest
Reviewed-on: https://go-review.googlesource.com/c/go/+/560515
Reviewed-by: Roland Shoemaker <roland@golang.org>
Auto-Submit: Bryan Mills <bcmills@google.com>
LUCI-TryBot-Result: Go LUCI <golang-scoped@luci-project-accounts.iam.gserviceaccount.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
fuzz Issues related to native fuzzing support NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. ToolSpeed
Projects
None yet
Milestone
Backlog
Development

No branches or pull requests
2 participants
@bcmills @gopherbot