cc @golang/fuzzing

Sorry, I'm still not sure I understand this.

• In your last example, you're calling F.Add twice with each input. Why? If the extra name given to F.Run is useful in the output, why isn't it useful in both places?
• Why test seed input values before calling F.Add? F.Fuzz could check them instead? I suppose you'd only have expected values for the seed inputs, but then why not just have a separate unit test?
• Not sure I see the parallelism advantage here at all. The top-level fuzz target just needs to call F.Add for each input, then F.Fuzz. It probably shouldn't do much computation other than that. The fuzz function passed to F.Fuzz may call T.Parallel, and that will run the seed inputs in parallel. It has no effect when fuzzing though; multiple inputs will run in parallel in different processes regardless of whether T.Parallel was called.

I'd expect this to be written as:

package atoi

import (
	"strconv"
	"testing"
)

var atoiTests = []struct {
	name, s  string
	expected int
}{
	{"zero", "0", 0},
	{"one", "1", 1},
}

func TestAtoi(t *testing.T) {
	for _, test := range atoiTests {
		test := test
		t.Run(test.name, func(t *testing.T) {
			t.Parallel()
			n, err := strconv.Atoi(test.s)
			if err != nil {
				t.Fatal(err)
			}
			if n != test.expected {
				t.Errorf("got %d; want %d", n, test.expected)
			}
		})
	}
}

func FuzzAtoi(f *testing.F) {
	for _, test := range atoiTests {
		f.Add(test.s)
	}
	f.Fuzz(func(t *testing.T, s string) {
		t.Parallel()
		n, err := strconv.Atoi(s)
		if err != nil {
			return
		}
		got := strconv.Itoa(n)
		if s != got {
			t.Fatalf("converting %q to integer and back, got %q", s, got)
		}
	})
}

In your last example, you're calling F.Add twice with each input. Why?

I tried to explain that in the text: "f.Add could be called in both places - inside and outside of a subtest. So it should be made thread-safe." It should not be, of course, but the idea was to show it either place can be picked by a user, and any of them will not be a mistake.

Why test seed input values before calling F.Add? F.Fuzz could check them instead? I suppose you'd only have expected values for the seed inputs, but then why not just have a separate unit test?

I guess that's the core of it – I don't want a separate unit test. The real package I'm working on is a data marshaller / unmarshaller. The unit tests for correct data looked almost identical to fuzz tests (unmarshal bytes into an object, re-marshal this object, compare bytes), so I merged them, as this example tried to show. Fuzzing draft proposal says talks about it: "In the long term, this design could start to replace existing table tests, seamlessly integrating into the existing Go testing ecosystem." Maybe I'm just living in the future.

Not sure I see the parallelism advantage here at all.

If you think about that part as a unit test, then parallelism advantage is clear.

I'd expect this to be written as

Note that your example:

• contains similar code in TestAtoi and FuzzAtoi (and the amount of shared code is larger in real case);
• contains a shared package-level variable.

My proposal allows one to write a single test function that both tests happy cases and feeds them to the fuzzer. Then the only tests that should be written separately are tests for returned error values.

Thanks for explaining, I think I understand better what you're getting at.

This would be a pretty big design change, so I'd like to make sure @katiehockman and @rolandshoemaker have a voice in this, too. Katie's on vacation for a few more weeks, so we won't be able to make a decision right away.

My API design intuition is that it's better to keep testing.F simpler and focused on fuzzing, even if it could be adapted to do most of what testing.T can do in non-fuzz tests. There's definitely some overlap though.