Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: per-subtest setup and cleanup #59291

Open
unkeep opened this issue Mar 28, 2023 · 9 comments
Open

proposal: testing: per-subtest setup and cleanup #59291

unkeep opened this issue Mar 28, 2023 · 9 comments
Labels
Milestone

Comments

@unkeep
Copy link

unkeep commented Mar 28, 2023

When writing tests in BDD style using subtests it's often necessary to invoke some common code before or after a subtest. In the example below we call beforeRun in the first place of subtests 1 and 2, otherwise the subtests would affect the outcome of each other:

type CoffeeMachine struct {
	on bool
}

func (a *CoffeeMachine) TurnOn() error {
	if a.on {
		return fmt.Errorf("aready on")
	}
	a.on = true
	return nil
}

func (a *CoffeeMachine) MakeCoffee() error {
	if !a.on {
		return fmt.Errorf("power is off")
	}

	return nil
}

func TestCoffeeMachine(t *testing.T) {
	// 0
	t.Run("given a new coffee machine", func(t *testing.T) {
		var a CoffeeMachine

		beforeRun := func(t *testing.T) {
			a = CoffeeMachine{}
		}

		// 1
		t.Run("it could be turned on", func(t *testing.T) {
			beforeRun(t)

			err := a.TurnOn()
			assert.NoError(t, err)

			// 1.1
			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err)
			})

			// 1.2
			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NoError(t, err)
			})
		})

		// 2
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			beforeRun(t)

			err := a.MakeCoffee()
			assert.Error(t, err)
		})
	})
}

It would be nice instead, if the testing package could invoke a specific function by itself before/after direct subtests of the parent where it's configured:

func TestCoffeeMachine(t *testing.T) {
	// 0
	t.Run("given a new coffee machine", func(t *testing.T) {
		var a CoffeeMachine

		// a new proposed method for testing.T
		t.BeforeRun(func(t *testing.T) {
			a = CoffeeMachine{}
		})

		// 1
		t.Run("it could be turned on", func(t *testing.T) {
			// the functor passed to BeforeRun is called before we reach this point

			err := a.TurnOn()
			assert.NoError(t, err)

			// 1.1
			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err)
			})

			// 1.2
			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NoError(t, err)
			})
		})

		// 2
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			// the functor passed to BeforeRun is called before we reach this point

			err := a.MakeCoffee()
			assert.Error(t, err)
		})
	})
}

The points against of the explicit beforeRun(t) calls from subtests are the following:

  1. it can simply be skipped by mistake
  2. it can be called from a wrong level by mistake

So I propose to add func (t *T) BeforeRun(func(t *testing.T) and func (t *T) AfterRun(func(t *testing.T) methods which should be invoked by the testing package before/after direct subtests of the parent where it's configured.

Relates to #39222 #40984

@gopherbot gopherbot added this to the Proposal milestone Mar 28, 2023
@ianlancetaylor
Copy link
Contributor

You can get a similar effect today by writing code like

func TestCounter(t *testing.T) {
    var a CoffeeMachine
    wrapper := func(f func(t *testing.T)) func(t *testing.T) {
        return func(t *testing.T) {
            a = CoffeeMachine{}
            f(t)
        }
    }
    t.Run("on", wrapper(func(t *testing.T) {
        err := a.TurnOn()
        ...
    }))
}

You can ensure that wrapper is always called by creating a table of test functions and then writing a loop to call t.Run for each one.

There is probably room for more control over tests and subtests, but in my opinion this proposal is too specific, adding functionality that will rarely be used.

@unkeep
Copy link
Author

unkeep commented Mar 28, 2023

@ianlancetaylor, yes, I considered a wrapper like yours. But in fact it does not resolve the same points against direct setup in subtests: you can forget to wrap and you can wrap at wrong level.
And I do not agree that the functionality will be rarely used, since having it in the testing package would help to get rid of dependencies such Ginkgo and Goconvey which are quite popular.

@unkeep unkeep closed this as completed Mar 28, 2023
@unkeep
Copy link
Author

unkeep commented Mar 28, 2023

Closed by mistake, sorry

@unkeep unkeep reopened this Mar 28, 2023
@willfaught
Copy link
Contributor

you can forget to wrap and you can wrap at wrong level.

It seems like you could equally forget to use BeforeRun.

Can you demonstrate what you mean by wrap at the wrong level?

@unkeep
Copy link
Author

unkeep commented Mar 28, 2023

It seems like you could equally forget to use BeforeRun.

Once you set BeforeRun, you can be sure that it will be called for a new added subtests of the same parent.

Can you demonstrate what you mean by wrap at the wrong level?

I meant to call wrap at other level of subtests than the level where it’s defined (deeper than necessary)

@nerock
Copy link

nerock commented Apr 6, 2023

@willfaught I think your objection could also apply to defer but it's something really useful in the language, and this could cover a necessity as @unkeep mentioned currently you have to use an external library or wrap every test function as other comment suggested

@dnephin
Copy link
Contributor

dnephin commented Apr 6, 2023

The functionality provided by the testing package today makes it quite easy to accomplish this style of testing. Here's an another example similar to the one above, which I think may do a better job of addressing the problems identified in the PR description.

func TestCoffeeMachine(t *testing.T) {
	givenCoffeeMachine := func(fn func(*testing.T, CoffeeMachine)) {
		m := CoffeeMachine{}
		t.Run("given a new coffee machine", func(t *testing.T) {
			// t.Cleanup( ... ) can be done here
			fn(t, m)
		})
	}

	givenCoffeeMachine(func(t *testing.T, a CoffeeMachine) {
		t.Run("it could be turned on", func(t *testing.T) {
			err := a.TurnOn()
			assert.NilError(t, err)

			t.Run("it could not be turned on again", func(t *testing.T) {
				err := a.TurnOn()
				assert.Error(t, err, "already on")
			})

			t.Run("it makes a coffee", func(t *testing.T) {
				err := a.MakeCoffee()
				assert.NilError(t, err)
			})
		})
	})

	givenCoffeeMachine(func(t *testing.T, a CoffeeMachine) {
		t.Run("MakeCoffee fails because it's not turned on", func(t *testing.T) {
			err := a.MakeCoffee()
			assert.Error(t, err, "power is off")
		})
	})
}

This approach avoids the shared a variable, and instead gives each call to givenCoffeeMachine a reference to the CoffeeMachine. This makes it impossible to forget because you wouldn't have a reference to CoffeeMachine without it. It also makes it very obvious when you're calling it at the wrong level because the function name is completely different, and you're getting a new reference that shadows the previous one.

@cespare
Copy link
Contributor

cespare commented Apr 6, 2023

What can be accomplished with t.AfterRun that cannot be accomplished today with t.Cleanup today?

@berlin-ab
Copy link

berlin-ab commented Sep 19, 2023

I was playing around with the suggestion provided by @dnephin above. Here's my example:


func TestExample(t *testing.T) {
	var value int
	var cleanupValue int

	setup := func(t *testing.T) {
		value = 10
		cleanupValue = 10
	}

	cleanup := func() {
		cleanupValue = 100
	}

	test := func(t *testing.T, message string, f func(*testing.T)) {
		t.Run(message, func(t *testing.T) {
			setup(t)
			t.Cleanup(cleanup)
			f(t)
		})
	}

	test(t, "it does something", func(t *testing.T) {
		setup := func(*testing.T) {
			value += 1
			require.Equal(t, 11, value)
		}

		cleanup := func() {
			cleanupValue = 200
		}

		test := func(t *testing.T, message string, f func(*testing.T)) {
			t.Run(message, func(t *testing.T) {
				setup(t)
				t.Cleanup(cleanup)
				f(t)
			})
		}

		t.Run("something", func(t *testing.T) {
			require.Equal(t, 10, value)
		})

		test(t, "it sub tests", func(t *testing.T) {
			require.Equal(t, 11, value)
		})

		t.Run("it cleans up using the subtest cleanup rule", func(t *testing.T) {
			require.Equal(t, 200, cleanupValue)
		})
	})

	test(t, "it does something else", func(t *testing.T) {
		t.Run("test", func(t *testing.T) {
			require.Equal(t, 10, value)
		})
	})

	t.Run("it cleans up using the toplevel cleanup rule", func(t *testing.T) {
		require.Equal(t, 100, cleanupValue)
	})
}

(edited to use defer for cleanup instead of t.Cleanup)

While the above works just fine, it also is a pattern that needs to be copied around a codebase and might drift overtime. Having a solution in the standard library would mostly prevent drift.

I've also been working on a small library that wraps testing to provide these setup/teardown hooks to solve the drift problem:

https://github.com/berlin-ab/testsuite

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

8 participants