cmd/go: -run= not working with fuzz subtest names #48149

rsc · 2021-09-02T13:04:36Z

% rm -r testdata/corpus && gotip test -run=XXX -fuzz=Fuzz'$'
found a crash, minimizing...
gathering baseline coverage, elapsed: 1.0s, workers: 16, left: 1948
--- FAIL: Fuzz (1.48s)
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xabfYo -->\n\n ^so --6|Zu   <!-- foo -->\n"
                out: "<p>\x02 <!-- \xab\xabfYo --></p>\n<p>^so --6|Zu   <!-- foo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xabfYo --&gt;</p>\n<p>^so --6|Zu   <!-- foo --></p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n\n ^so --6|Zu"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n<p>^so --6|Zu</p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n<p>^so --6|Zu</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n\n ^s"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n<p>^s</p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n<p>^s</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->\n"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02 <!-- \xab\xb1fYo -->"
                out: "<p>\x02 <!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02 &lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!-- \xab\xb1fYo -->"
                out: "<p>\x02<!-- \xab\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!-- \xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xab\xb1fYo -->"
                out: "<p>\x02<!--\xab\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!--\xab\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1fYo -->"
                out: "<p>\x02<!--\xb1fYo --></p>\n"
                gout: "<p>\x02&lt;!--\xb1fYo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1Yo -->"
                out: "<p>\x02<!--\xb1Yo --></p>\n"
                gout: "<p>\x02&lt;!--\xb1Yo --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1o -->"
                out: "<p>\x02<!--\xb1o --></p>\n"
                gout: "<p>\x02&lt;!--\xb1o --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1 -->"
                out: "<p>\x02<!--\xb1 --></p>\n"
                gout: "<p>\x02&lt;!--\xb1 --&gt;</p>\n"
        --- FAIL: Fuzz (0.00s)
            fuzz_test.go:60: in: "\x02<!--\xb1-->"
                out: "<p>\x02<!--\xb1--></p>\n"
                gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
    
    Crash written to testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83
    To re-run:
    go test rsc.io/markdown -run=Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83
FAIL
exit status 1
FAIL	rsc.io/markdown	1.638s

Bug 1t: The re-run command should not include the package name if I didn't include the package name. That is, if I ran go test and not go test <args>, then it should say go test -run... not go test <pkgname> -run.... It's jarring to see the package name appear when I wasn't using it explicitly.

Bug 2: The -run= argument does not match the test failure:

r% gotip test -run=Fuzz
--- FAIL: Fuzz (0.06s)
    --- FAIL: Fuzz/testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 (0.00s)
        fuzz_test.go:60: in: "\x02<!--\xb1-->"
            out: "<p>\x02<!--\xb1--></p>\n"
            gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
FAIL
exit status 1
FAIL	rsc.io/markdown	0.189s

Note that the failure has an extra 'testdata/corpus' in its name. I assume that was meant to be left out of the name printed by the test.

Bug 3: The -run argument does not work to select subtests of fuzz targets:

r% gotip test -run=Fuzz/somethingelse
--- FAIL: Fuzz (0.06s)
    --- FAIL: Fuzz/testdata/corpus/Fuzz/c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 (0.00s)
        fuzz_test.go:60: in: "\x02<!--\xb1-->"
            out: "<p>\x02<!--\xb1--></p>\n"
            gout: "<p>\x02&lt;!--\xb1--&gt;</p>\n"
FAIL
exit status 1
FAIL	rsc.io/markdown	0.180s

Saying Fuzz/somethingelse should have matched somethingelse against testdata and not matched. Or the implication of the suggested command is that it should have matched somethingelse against c1c09817115370fcbcf4ea60d47a71bdd843fbe249bb987f98a9421b1543de83 and still not matched. Either way, the test should not have run, but it did.

/cc @golang/fuzzing

The text was updated successfully, but these errors were encountered:

katiehockman · 2021-09-14T21:37:07Z

Bug 2: The -run= argument does not match the test failure:
...
Note that the failure has an extra 'testdata/corpus' in its name. I assume that was meant to be left out of the name printed by the test.

@rsc Can you clarify what you mean here? I don't see the bug. Are you just noting that it says 'testdata/corpus' (now 'testdata/fuzz') in the fail line? That's because it corresponds to the corpus file written in 'testdata/corpus'

katiehockman · 2021-10-06T20:34:13Z

Ah I think I understand what @rsc is saying now. Correct me if I'm wrong, but it sounds like the confusion is because the test is runnable with -run=Fuzz/c1c09817... but the output in the terminal is Fuzz/testdata/fuzz/Fuzz/c1c0981711537... when it should be Fuzz/c1c09817..

Will work on these.

katiehockman · 2021-10-07T15:24:19Z

Bug 3 is going to be a bit difficult to fix, but not impossible.

The trickiest bit is that we have to load the corpus in order to know which seed corpus entries (if any) that the -run argument matches. i.e. we don't know what's in testdata/fuzz until we've loaded that from disk. Right now, we load the corpus in f.Fuzz since that's when we can do things like check that the types match. However, by the time we're in f.Fuzz, it's too late for us to know whether or not any of the seed corpus is going to run. By that point, we've already printed === RUN (in runFuzzTargets) and fRunner has been executed, which will set the tests to ran.
What that means is that we could get something like this:

go test -v -run FuzzFoo/nomatchhere
=== RUN   FuzzFoo
--- PASS: FuzzFoo (0.00s)
PASS
ok      foo      0.133s

When in reality it should look like this:

go test -v -run FuzzFoo/nomatchhere   
testing: warning: no tests to run
PASS
ok      foo      0.617s [no tests to run]

One possible solution is to load all of the seed corpus in runFuzzTargets, before fRunner is called. If it doesn't match anything, then we just don't execute fRunner.

Will keep looking into it.

katiehockman · 2021-10-07T17:44:37Z

This actually might not be as complicated as I thought. If we treat this the same way that we treat subtests with t.Run, then it's fine to print the === RUN and --- PASS lines even if it isn't executed. The important thing is to match the behavior of TestXxx functions with -run. I should be able to send a patch that fixes these issues shortly.

gopherbot · 2021-10-07T20:41:02Z

Change https://golang.org/cl/354632 mentions this issue: testing: fix -run behavior with fuzz tests

This change fixes some issues with -run, and the subsequent command line output when running in verbose mode. It replaces CorpusEntry.Name with CorpusEntry.Path, and refactors the code accordingly. This change also adds a lot of additional tests which check explicit command line output when fuzz targets are run without fuzzing. This will be important to avoid regressions. Updates #48149 Change-Id: If34b1f51db646317b7b51c3c38ae53231d01f568 Reviewed-on: https://go-review.googlesource.com/c/go/+/354632 Trust: Katie Hockman <katie@golang.org> Run-TryBot: Katie Hockman <katie@golang.org> TryBot-Result: Go Bot <gobot@golang.org> Reviewed-by: Jay Conrod <jayconrod@google.com>

katiehockman · 2021-10-14T17:48:45Z

I'm not actually sure how to fix Bug 1t.

The goal of that is to only print the package name in the To re-run: go test <pkg> -run=FuzzFoo/dba83... line if the package was provided in the original go test -fuzz command. But as far as I can tell, we only know what package arguments were provided at compile time.
Passing it all the way from the cmd/go/internal/test package to the testing package is probably possible, but will be messy. I'm not sure it's worth it.

rsc · 2021-11-08T15:36:37Z

For 1t (the t appears to be for typo), I would suggest just dropping the pkg path unconditionally. The common case is testing a single package. People are smart enough to understand they need to add it back if they were running test on multiple packages.

gopherbot · 2021-11-08T17:27:06Z

Change https://golang.org/cl/362116 mentions this issue: testing: remove package from fuzz crash message

rsc added the release-blocker label Sep 2, 2021

rsc added this to the Go1.18 milestone Sep 2, 2021

cherrymui added the NeedsFix The path to resolution is known, but the work has not been done. label Sep 2, 2021

jayconrod added the fuzz Issues related to native fuzzing support label Sep 9, 2021

katiehockman added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 28, 2021

katiehockman removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Oct 6, 2021

katiehockman added the okay-after-beta1 Used by release team to mark a release-blocker issue as okay to resolve either before or after beta1 label Oct 20, 2021

gopherbot closed this as completed in 2559a98 Nov 9, 2021

golang locked and limited conversation to collaborators Nov 9, 2022

gopherbot added the FrozenDueToAge label Nov 9, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: -run= not working with fuzz subtest names #48149

cmd/go: -run= not working with fuzz subtest names #48149

rsc commented Sep 2, 2021

katiehockman commented Sep 14, 2021

katiehockman commented Oct 6, 2021

katiehockman commented Oct 7, 2021

katiehockman commented Oct 7, 2021

gopherbot commented Oct 7, 2021

katiehockman commented Oct 14, 2021

rsc commented Nov 8, 2021

gopherbot commented Nov 8, 2021

cmd/go: -run= not working with fuzz subtest names #48149

cmd/go: -run= not working with fuzz subtest names #48149

Comments

rsc commented Sep 2, 2021

katiehockman commented Sep 14, 2021

katiehockman commented Oct 6, 2021

katiehockman commented Oct 7, 2021

katiehockman commented Oct 7, 2021

gopherbot commented Oct 7, 2021

katiehockman commented Oct 14, 2021

rsc commented Nov 8, 2021

gopherbot commented Nov 8, 2021