Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: go test -fuzz structured output #68071

Open
prototype0372 opened this issue Jun 19, 2024 · 1 comment
Open

proposal: go test -fuzz structured output #68071

prototype0372 opened this issue Jun 19, 2024 · 1 comment
Labels
Milestone

Comments

@prototype0372
Copy link

Proposal Details

This is my first issue in the Go repo - if I miss something out please let me know and I'll add it. Furthermore, if this problem is solved then please point me in the right direction. TIA!

Problem to solve

Do some post-processing on fuzzing output as part of CI.

Current behaviour

There is a -json flag that may be passed for fuzz tests that does a fairly basic job of outputting the results of the test in a line-by-line json stream. For example

{"Time":"2024-06-19T14:53:25.086130955+01:00","Action":"run","Package":"example.com/foo/bar","Test":"FuzzFooBar"}
{"Time":"2024-06-19T14:53:25.086133564+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"=== RUN   FuzzFooBar\n"}
{"Time":"2024-06-19T14:53:25.098551229+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"fuzz: elapsed: 0s, gathering baseline coverage: 0/112 completed\n"}
{"Time":"2024-06-19T14:53:25.308440206+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"fuzz: elapsed: 0s, gathering baseline coverage: 112/112 completed, now fuzzing with 20 workers\n"}
{"Time":"2024-06-19T14:53:27.05786798+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"fuzz: elapsed: 2s, execs: 93141 (47539/sec), new interesting: 0 (total: 112)\n"}
{"Time":"2024-06-19T14:53:27.058004081+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"--- FAIL: FuzzFooBar (1.97s)\n"}
{"Time":"2024-06-19T14:53:27.0580113+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"    --- FAIL: FuzzFooBar (0.00s)\n"}
{"Time":"2024-06-19T14:53:27.058015058+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"        testing.go:1590: panic: =0\n"}
{"Time":"2024-06-19T14:53:27.058018557+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            goroutine 6081 [running]:\n"}
{"Time":"2024-06-19T14:53:27.058022151+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            runtime/debug.Stack()\n"}
{"Time":"2024-06-19T14:53:27.058025292+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/runtime/debug/stack.go:24 +0x9b\n"}
{"Time":"2024-06-19T14:53:27.058028825+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            testing.tRunner.func1()\n"}
{"Time":"2024-06-19T14:53:27.058031564+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/testing/testing.go:1590 +0x1c8\n"}
{"Time":"2024-06-19T14:53:27.058034268+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            panic({0x8a4a80?, 0xc007376420?})\n"}
{"Time":"2024-06-19T14:53:27.058037553+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/runtime/panic.go:770 +0x132\n"}
{"Time":"2024-06-19T14:53:27.058040183+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            example.com/foo/bar.parseFoo({0xc00731d195, 0x5}, 0x0)\n"}
{"Time":"2024-06-19T14:53:27.058042916+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/home/user/go/src/example.com/foo/bar/baz.go:81 +0x3aa\n"}
{"Time":"2024-06-19T14:53:27.058045831+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            example.com/foo/bar.FooBar({0xc00731d195?, 0x4?})\n"}
{"Time":"2024-06-19T14:53:27.058048718+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/home/user/go/src/example.com/foo/bar/baz.go:50 +0x3d\n"}
{"Time":"2024-06-19T14:53:27.058051449+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            example.com/foo/bar_test.FuzzFooBar.func1(0xc007402340, {0xc00731d195?, 0x0?})\n"}
{"Time":"2024-06-19T14:53:27.058056846+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/home/user/go/src/example.com/foo/bar/baz_test.go:209 +0x27\n"}
{"Time":"2024-06-19T14:53:27.058060045+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            reflect.Value.call({0x8a9a00?, 0x9502d0?, 0x13?}, {0x91e6ec, 0x4}, {0xc00737c6c0, 0x2, 0x2?})\n"}
{"Time":"2024-06-19T14:53:27.058063124+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/reflect/value.go:596 +0xca6\n"}
{"Time":"2024-06-19T14:53:27.058066795+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            reflect.Value.Call({0x8a9a00?, 0x9502d0?, 0x55686d?}, {0xc00737c6c0?, 0x91dba0?, 0xf?})\n"}
{"Time":"2024-06-19T14:53:27.058070343+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/reflect/value.go:380 +0xb9\n"}
{"Time":"2024-06-19T14:53:27.058073562+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            testing.(*F).Fuzz.func1.1(0xc007402340?)\n"}
{"Time":"2024-06-19T14:53:27.058094739+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/testing/fuzz.go:335 +0x325\n"}
{"Time":"2024-06-19T14:53:27.058117+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            testing.tRunner(0xc007402340, 0xc00734dd40)\n"}
{"Time":"2024-06-19T14:53:27.058125926+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/testing/testing.go:1689 +0xfb\n"}
{"Time":"2024-06-19T14:53:27.058131209+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            created by testing.(*F).Fuzz.func1 in goroutine 6\n"}
{"Time":"2024-06-19T14:53:27.058135749+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \t/snap/go/10630/src/testing/fuzz.go:322 +0x574\n"}
{"Time":"2024-06-19T14:53:27.058140371+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"            \n"}
{"Time":"2024-06-19T14:53:27.058146352+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"    \n"}
{"Time":"2024-06-19T14:53:27.05815182+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"    Failing input written to testdata/fuzz/FuzzFooBar/7cad059f4ce4e35f\n"}
{"Time":"2024-06-19T14:53:27.058156412+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"    To re-run:\n"}
{"Time":"2024-06-19T14:53:27.058160946+01:00","Action":"output","Package":"example.com/foo/bar","Test":"FuzzFooBar","Output":"    go test -run=FuzzFooBar/7cad059f4ce4e35f\n"}
{"Time":"2024-06-19T14:53:27.058167091+01:00","Action":"fail","Package":"example.com/foo/bar","Test":"FuzzFooBar","Elapsed":1.97}
{"Time":"2024-06-19T14:53:27.05818611+01:00","Action":"output","Package":"example.com/foo/bar","Output":"FAIL\n"}
{"Time":"2024-06-19T14:53:27.073721916+01:00","Action":"output","Package":"example.com/foo/bar","Output":"exit status 1\n"}
{"Time":"2024-06-19T14:53:27.073757334+01:00","Action":"output","Package":"example.com/foo/bar","Output":"FAIL\texample.com/foo/bar\t1.992s\n"}
{"Time":"2024-06-19T14:53:27.073765388+01:00","Action":"fail","Package":"example.com/foo/bar","Elapsed":1.9929999999999999}

The particular field I'm interested in is Output as this contains the really important information from a testers perspective. Currently this field is just a copy of whatever line would otherwise have been printed to stdout if -json had been omitted.

Current workarounds

It's not difficult to do some regex matching on the Output field to pull out the interesting information (full stacktrace and sha of the failing input), but it feels like there could be a better way. And I don't know whether the text on stdout is considered a stable API.

Proposal

Make this Output field a structured type, or add new structured fields so that post-processing is as easy as running some jq. Something as simple as

type FuzzFail struct {
    Stacktrace []string
    FailingInputSHA string
    FailingInput []byte
}

would be ideal. Even having the FailingInput []byte in there is optional since I can pull that information out from testdata/fuzz given the sha and test name.

@gopherbot gopherbot added this to the Proposal milestone Jun 19, 2024
@ianlancetaylor ianlancetaylor moved this to Incoming in Proposals Jun 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

3 participants