Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing: fuzzer aborts with "fuzzing process hung or terminated unexpectedly while minimizing: EOF" #52569

Open
lmb opened this issue Apr 26, 2022 · 8 comments
Open

testing: fuzzer aborts with "fuzzing process hung or terminated unexpectedly while minimizing: EOF" #52569

lmb opened this issue Apr 26, 2022 · 8 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Unplanned

Comments

@lmb
Copy link
Contributor

lmb commented Apr 26, 2022 •
edited
Loading

What version of Go are you using (go version)?

go version go1.18.1 linux/arm64

Does this issue reproduce with the latest release?

Yes.

What operating system and processor architecture are you using (go env)?

go env Output
GO111MODULE=""
GOARCH="arm64"
GOBIN="/home/lorenz/.local/bin"
GOCACHE="/home/lorenz/.cache/go-build"
GOENV="/home/lorenz/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/lorenz/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/lorenz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_arm64"
GOVCS=""
GOVERSION="go1.18.1"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/lorenz/ebpf/go.mod"
GOWORK=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build640695223=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I ran the following in the https://github.com/cilium/ebpf repo at d4e048f8a6d8d213b452c61573bd8150c35b557c.

$ go test -fuzz FuzzSpec -run XXX ./internal/btf/
WARNING: Failed to adjust rlimit, tests may fail
fuzz: elapsed: 0s, gathering baseline coverage: 0/14 completed
fuzz: elapsed: 0s, gathering baseline coverage: 14/14 completed, now fuzzing with 4 workers
fuzz: elapsed: 3s, execs: 21471 (7155/sec), new interesting: 1 (total: 15)
fuzz: elapsed: 5s, execs: 22115 (279/sec), new interesting: 1 (total: 15)
--- FAIL: FuzzSpec (5.31s)
    fuzzing process hung or terminated unexpectedly while minimizing: EOF
    Failing input written to testdata/fuzz/FuzzSpec/a2142308b78fc52a699d1f6e6c6e6552ed94b5ff5c376733a080dfb12fd98cd4
    To re-run:
    go test -run=FuzzSpec/a2142308b78fc52a699d1f6e6c6e6552ed94b5ff5c376733a080dfb12fd98cd4
FAIL
exit status 1
FAIL	github.com/cilium/ebpf/internal/btf	5.324s

What did you expect to see?

I expected the command given to reproduce the issue to result in a crash.

What did you see instead?

go test -run=FuzzSpec/a2142308b78fc52a699d1f6e6c6e6552ed94b5ff5c376733a080dfb12fd98cd4 -v
WARNING: Failed to adjust rlimit, tests may fail
testing: warning: no tests to run
PASS
ok  	github.com/cilium/ebpf	0.004s

The contents of the file are:

$ cat internal/btf/testdata/fuzz/FuzzSpec/a2142308b78fc52a699d1f6e6c6e6552ed94b5ff5c376733a080dfb12fd98cd4
go test fuzz v1
[]byte("\x9f\xeb\x01\x00\x18\x00\x00\x000000000000000\x00\x009")
@lmb lmb mentioned this issue Apr 26, 2022
Merged
@dr2chase dr2chase added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 8, 2022
@dr2chase
Copy link
Contributor

dr2chase commented May 8, 2022

@rolandshoemaker can you give this a look? I read this as the fuzzer found a sporadic issue that will not repeat itself on command, but maybe something else is going on.

@alexstan12
Copy link

I am experiencing the same behavior. The data included in testdata successfully passes the unit test, so data is not the issue.

@alexstan12 alexstan12 mentioned this issue Jun 20, 2022
Open
@klauspost
Copy link
Contributor

klauspost commented Jul 6, 2022 •
edited
Loading

Seeing the same issue.

It fails consistently, except when using -parallel=1. Obviously that is wasting 15 cores, so far from optimal.

Edit: Further investigation leads me to believe that the cause of this is an OOM condition. Limiting memory use appears to fix this issue.

@seankhliao seankhliao added this to the Unplanned milestone Aug 20, 2022
@dagood
Copy link
Contributor

dagood commented Sep 20, 2022 •
edited
Loading

Further investigation leads me to believe that the cause of this is an OOM condition. Limiting memory use appears to fix this issue.

How did you limit memory use? Do you mean reducing the memory the test itself uses, or some more general limit?

I'm also running into this error on windows-amd64 and linux-amd64 with a Go build based on f4becf1, recent master branch. Currently I'm running with GOMAXPROCS=1 GOGC=10 GOMEMLIMIT=1GiB and -parallel=1 (a wild guess based on this thread) and the error still occurs in seemingly the same way if I let the fuzz tests run for long enough.

(I haven't been able to reproduce on a machine I fully control yet, so I'm having a hard time monitoring what about my system might be causing crashes.)

Edit: Since posting, I've been able to reproduce the OOM killer causing this error on Linux by running with very little available memory in a VM. We've also been able to fix it by changing the fuzz tests themselves to use less memory. I was under the impression the memory usage of the fuzz test infra was the problem, but it seems (at least partially) under our control.

I do wish the error were a little clearer--EOF is one of those errors where at first glance it seems to be about files, but it could actually be just about anything.

ddworken added a commit to ddworken/hishtory that referenced this issue Sep 23, 2022
@ddworken
Improve copy function error checking + delete make fuzz because of go…
9895a5c 
…lang/go#52569
@b-scholz b-scholz mentioned this issue Apr 15, 2023
Merged
@belimawr
Copy link

I'm also seeing the same issue on Linux.

go version 

go version
go version go1.20.5 linux/amd64

go env 

go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/tiago/.cache/go-build"
GOENV="/home/tiago/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/tiago/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/tiago/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
GOAMD64="v1"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/dev/null"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -Wl,--no-gc-sections -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build362825651=/tmp/go-build -gno-record-gcc-switches"


=== RUN   FuzzFilestreamID
fuzz: elapsed: 0s, gathering baseline coverage: 0/41 completed
fuzz: elapsed: 3s, gathering baseline coverage: 30/41 completed
fuzz: elapsed: 4s, gathering baseline coverage: 41/41 completed, now fuzzing with 16 workers
fuzz: elapsed: 6s, execs: 67 (12/sec), new interesting: 0 (total: 41)
fuzz: elapsed: 9s, execs: 368 (100/sec), new interesting: 0 (total: 41)
fuzz: elapsed: 12s, execs: 383 (5/sec), new interesting: 0 (total: 41)
fuzz: elapsed: 15s, execs: 395 (4/sec), new interesting: 0 (total: 41)
fuzz: minimizing 33-byte failing input file
fuzz: elapsed: 18s, minimizing
fuzz: elapsed: 20s, minimizing
--- FAIL: FuzzFilestreamID (19.67s)
    fuzzing process hung or terminated unexpectedly while minimizing: EOF
    Failing input written to testdata/fuzz/FuzzFilestreamID/1c1ced2e8b2131bb
    To re-run:
    go test -run=FuzzFilestreamID/1c1ced2e8b2131bb
=== NAME  
FAIL
exit status 1
FAIL    github.com/elastic/beats/v7/filebeat/input/filestream   95.419s

I'm not quite sure I'm experiencing OOM, I run go test -fuzz while keeping an eye on htop and did not see a memory spike or the memory filling up.

@belimawr belimawr mentioned this issue Jun 29, 2023
Draft
6 tasks
@Dmytro-yakymuk Dmytro-yakymuk mentioned this issue Aug 3, 2023
Closed
@shogo82148 shogo82148 mentioned this issue Sep 11, 2023
Open
@AlekSi
Copy link
Contributor

AlekSi commented Jan 17, 2024

Might be related to #56238

@AlekSi AlekSi mentioned this issue Jan 17, 2024
Closed
@maxammann
Copy link

@lmb I'm supposing the EOF is related to the process getting killed due to OOM.

At least I experienced an OOM when running the example you posted initially:

fuzz: elapsed: 50s, execs: 9495 (0/sec), new interesting: 1 (total: 14)
fuzz: elapsed: 1m8s, execs: 9495 (0/sec), new interesting: 1 (total: 14)
fuzz: elapsed: 1m8s, execs: 9495 (0/sec), new interesting: 1 (total: 14)
[LWP 14680 exited]
[LWP 14679 exited]
[LWP 14678 exited]
[LWP 14677 exited]
[LWP 14676 exited]
[LWP 14675 exited]
[LWP 14673 exited]
[LWP 14674 exited]
[New process 14673]

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) fuzz: elapsed: 1m9s, execs: 11424 (3286/sec), new interesting: 2 (total: 15)
fuzz: elapsed: 1m12s, execs: 18743 (2440/sec), new interesting: 3 (total: 16)
fuzz: elapsed: 1m15s, execs: 18743 (0/sec), new interesting: 3 (total: 16)
fuzz: elapsed: 1m23s, execs: 18743 (0/sec), new interesting: 3 (total: 16)
fuzz: elapsed: 1m23s, execs: 18743 (0/sec), new interesting: 3 (total: 16)
fuzz: elapsed: 1m23s, execs: 19649 (5249/sec), new interesting: 3 (total: 16)
--- FAIL: FuzzSpec (83.18s)
    fuzzing process hung or terminated unexpectedly while minimizing: EOF
    Failing input written to testdata/fuzz/FuzzSpec/4ac8ead72d16ddcc
    To re-run:
    go test -run=FuzzSpec/4ac8ead72d16ddcc

[ 2014.895523] Out of memory: Killed process 14690 (btf.test) total-vm:6535944kB, anon-rss:3566968kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:8672kB oom_score_adj:0

@maxammann maxammann mentioned this issue Feb 16, 2024
Open
@derat derat mentioned this issue Mar 22, 2024
Open
adamdecaf added a commit to moov-io/metro2 that referenced this issue Apr 11, 2024
@adamdecaf
test/fuzz: force GOMEMLIMIT to cgroup

Unverified

This commit is not signed, but one or more authors requires that any commit attributed to them is signed.
Learn about vigilant mode
Loading
Loading status checks…
fab5936 
See: golang/go#52569
@ggrieco-tob ggrieco-tob mentioned this issue Jun 6, 2024
Merged
12 tasks
@AlekSi AlekSi mentioned this issue Jul 24, 2024
Closed
@shuLhan shuLhan mentioned this issue Feb 4, 2025
Closed
@gabyhelp gabyhelp mentioned this issue Mar 10, 2025
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
10 participants
@AlekSi @belimawr @lmb @maxammann @dr2chase @klauspost @seankhliao @dagood @alexstan12 and others