Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: test -fuzz gives a confusing error when types mismatch #48635

Closed
mvdan opened this issue Sep 26, 2021 · 6 comments
Closed

cmd/go: test -fuzz gives a confusing error when types mismatch #48635

mvdan opened this issue Sep 26, 2021 · 6 comments
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@mvdan
Copy link
Member

mvdan commented Sep 26, 2021

I wrote a fuzz func like:

f.Fuzz(func(t *testing.T, src string, parserOpts uint8, printerOpts uint16) {

Then I used f.Add as follows:

f.Add("{ (foo; bar); baz; }", 0, 0)

I got a fairly confusing error:

fuzz_test.go:66: mismatched types in corpus entry: [echo foo 0 0], want [string uint8 uint16]

Intuitively, I thought my code was right, because the untyped constant 0 can be assigned to uint8 and uint16 just fine. It took me a minute to realise that, when passing 0 as an argument, it assumes the int type.

I think the following error message would be much easier to understand:

fuzz_test.go:66: mismatched types in corpus entry: [string int int], want [string uint8 uint16]

or, if you want to include the values:

fuzz_test.go:66: mismatched types in corpus entry: [string("echo foo") int(0) int(0)], want [string uint8 uint16]
@mvdan
Copy link
Member Author

mvdan commented Sep 26, 2021

Another reason the current error is confusing is that [echo foo 0 0] looks like four arguments, when it's actually three - due to the lack of quoting.

@mvdan mvdan added the fuzz Issues related to native fuzzing support label Sep 28, 2021
@mvdan
Copy link
Member Author

mvdan commented Sep 28, 2021

(forgot to cc @katiehockman @jayconrod on this one, doing so quietly during the quiet week...)

@mknyszek mknyszek added this to the Go1.18 milestone Oct 4, 2021
@mknyszek mknyszek added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Oct 4, 2021
@cespare
Copy link
Contributor

cespare commented Nov 3, 2021

I just ran into this as well. I hope the message can be improved before 1.18; I suspect that many people are going to see it when they use fuzzing for the first time.

In my case, I had something like this:

f.Add("")
f.Fuzz(func(t *testing.T, b []byte) { ... })

which led to this confusing message:

fuzzdemo_test.go:9: mismatched types in corpus entry: [], want [[]uint8]

Simply adding quotes helps a lot:

mismatched types in corpus entry: [""], want [[]uint8]

Both of @mvdan's suggestions are better yet:

mismatched types in corpus entry: [string], want [[]uint8]
mismatched types in corpus entry: [string("")], want [[]uint8]

@katiehockman katiehockman self-assigned this Nov 3, 2021
@katiehockman katiehockman added NeedsFix The path to resolution is known, but the work has not been done. and removed NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels Nov 3, 2021
@gopherbot
Copy link

Change https://golang.org/cl/361114 mentions this issue: internal/fuzz: improve error for mismatched types

@katiehockman
Copy link
Contributor

Thanks @mvdan for the report, and @cespare for the update. I had missed this issue during triage!

I opted for the original suggestion of fuzz_test.go:66: mismatched types in corpus entry: [string int int], want [string uint8 uint16]. That seems the easiest to understand.

@mvdan
Copy link
Member Author

mvdan commented Nov 3, 2021

Thank you! Certainly a simple change, but will be nice for the UX :)

@golang golang locked and limited conversation to collaborators Jun 23, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge fuzz Issues related to native fuzzing support NeedsFix The path to resolution is known, but the work has not been done.
Projects
Status: No status
Development

No branches or pull requests

5 participants