Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

testing/quick: add support for shrinking #9282

Open
ghost opened this issue Dec 12, 2014 · 5 comments
Open

testing/quick: add support for shrinking #9282

ghost opened this issue Dec 12, 2014 · 5 comments
Labels
Suggested Issues that may be good for new contributors looking for work to do.
Milestone
Unplanned

Comments

@ghost
Copy link

ghost commented Dec 12, 2014

The testing/quick package automatically generates test cases to be checked against a property. However, it often generates complicated values which obscure the source of test failures. Shrinking is a feature from the original QuickCheck which eases this burden by "simplifying" a failed test case as much as possible.

Evan Shaw gave a presentation on testing/quick at Gopher Summerfest, and started work on adding shrinking to the package in the stdlib. I could't find any official activity related to it, though.

I think he got a good start on the problem, but I also think the interface definition for Shrinker would be significantly simpler like this:

// A Shrinker can produce simpler values with its same type.
type Shrinker interface {
    // Shrink returns a closure which successively yields simpler values of its type.
    Shrink() func() reflect.Value
}

Also, because testing/quick generates random values for its test cases, I believe the behavior of quick.Check() can be changed to support shrinking without breaking the Go compatibility guidelines.

I can work on this feature if the Go team decides it's a good idea.
@ianlancetaylor ianlancetaylor added repo-main Suggested Issues that may be good for new contributors looking for work to do. labels Dec 12, 2014
@josharian
Copy link
Contributor

cc @arschles

@rsc rsc added this to the Unplanned milestone Apr 10, 2015
@spacejam
Copy link

testing/quick stands to benefit greatly from this functionality. As it stands, you can't use it for automatic generation of regression tests, as other quickcheck implementations allow for with shrinking. Without shrinking, it is so much more cumbersome to figure out what actually caused an issue.

Many people understand quickcheck's utility for testing things like decoder(encoder(val)) == val and for this, testing/quick is adequate, but when shrinking enters the equation, the testing library becomes far more powerful, and capable of testing complex interactions. You can do things like generate a slice of events/operations/requests to apply to a system, and if the property fails to hold, the shrinker can:

  1. drop out individual elements of the slice
  2. re-run the property test on the shrunken input
  3. if the shrunken slice still fails, GOTO 1
  4. if dropping more elements fails to cause the property to be violated, we have probably just found a concise, more easily debuggable sequence of events/operations/requests that will trip up your system.

I really like using other quickcheck implementations to generate a sequence of external requests and network connectivity issues that then get fed into a simulated distributed system with simulated network + clocks. When the generated input fails, the shrinker reduces it dramatically, and the (potentially quite complex) issue becomes far more understandable.

Further, shrinkers automatically generate regression tests for you! Turn the failing non-deterministic sequence found by quickcheck into a regression test for deterministically watching for similar failures in the future.

Does anyone NOT think that would make a great gophercon talk? :P

@spacejam
Copy link

I'd like to volunteer to build this, since it's been a few years since (deleted OP) proposed and volunteered.

Seriously, this is so useful that sometimes I get away with only writing a single test for a low-impact interactive system, and after shrinking the testing library just tells me what I need to fix with reasonable thoroughness and clarity.

What does the team think?

@ianlancetaylor
Copy link
Contributor

Thanks for looking into this.

What does the team think?

I think I would like to see a specific API proposal.

@spacejam spacejam mentioned this issue Dec 14, 2017
Closed
@spacejam
Copy link

@ianlancetaylor #23135

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Suggested Issues that may be good for new contributors looking for work to do.
Projects
None yet
Milestone
Unplanned
Development

No branches or pull requests
4 participants
@josharian @rsc @spacejam @ianlancetaylor