Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: testing: output timing and outcome of TestMain or possible setup and teardown functions #57326

Open
doron-cohen opened this issue Dec 15, 2022 · 4 comments
Labels
Milestone

Comments

@doron-cohen
Copy link

Hi,

We are currently using the TestMain function to setup and teardown test-containers in our integration tests. This allows us to prepare the environment once for the entire test suite and save time.
We also monitor our test CI step by using gotestsum which practically parses the output from go test -json. This lets us keep the CI snappy.

One issue we are having is that there is no way to monitor TestMain's timing or outcome. In gotestsum there is some code to add a fake test case in the case the failure occurred in TestMain (see comment by @dnephin ). But, there is still no way to monitor how long did the logic before and after the tests ran.

There are several possible ways to tackle this IMO:

Declaring setup and teardown functions

Introduce new functions for setup and teardown:

func TestMain(m *testing.M) {
	m.Setup(setupFunc)
	m.Teardown(teardownFunc)
}

This can work just like testing.T.Cleanup does. I think this is beneficial for other use cases where there is a need to better organize testing code.

Emitted JSONs could look like:

{"Time":"2022-12-10T14:13:37.4857284-02:00","Action":"pass","Package":"example.com/cool/app","Function":"setupFunc","Elapsed":0.0321}

Monitor TestMain timing without m.Run time

Another option is to time the entire TestMain but stop the timer on m.Run and resume when it ends.

Hypothetical code:

func (m *testing.M) Run() int {
	m.timer.Stop()
	defer m.timer.Continue()

	# ... run tests
}

Emitted JSONs could look like:

{"Time":"2022-12-10T14:13:37.4857284-02:00","Action":"pass","Package":"example.com/cool/app","Function":"TestMain","Elapsed":10.3853}
@gopherbot gopherbot added this to the Proposal milestone Dec 15, 2022
@seankhliao
Copy link
Member

note testing.M.Cleanup was previously declined in #42161

@doron-cohen
Copy link
Author

note testing.M.Cleanup was previously declined in #42161

Thanks for the reference. I think there is another valid use case here which I didn't see in that proposal. Also, we can achieve TestMain timing without these as I suggested.

@kegsay
Copy link

kegsay commented Nov 16, 2023

note testing.M.Cleanup was previously declined in #42161

I believe this was erroneously declined, on the basis that:

It seems to me that testing.M.Cleanup can be implemented simply using defer.

This is not true. Consider:

package test

import (
	"fmt"
	"os"
	"testing"
)

func TestMain(m *testing.M) {
	defer func() {
		fmt.Println("cleanup code")
	}()
	os.Exit(m.Run())
}

func TestFoo(t *testing.T) {
	t.Fatalf("oh no")
}

Cleanup code does not execute because of os.Exit(), which is called because the docs tell you to:

Run runs the tests. It returns an exit code to pass to os.Exit.

Let's ignore that for now then. Consider this other code:

package test

import (
	"fmt"
	"testing"
)

func TestMain(m *testing.M) {
	defer func() {
		fmt.Println("cleanup code")
	}()
	m.Run()
}

func TestFoo(t *testing.T) {
	panic("oh no")
}

In this scenario, the defer does not run at all.

It's simply not true that you can get cleanup functions working at the testing.M level with the current tooling.

@adabuleanu
Copy link

It's simply not true that you can get cleanup functions working at the testing.M level with the current tooling.

@kegsay argument is true. I experienced the same behavior. Any progress on this feature or any alternatives on working around this scenario?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
Status: Incoming
Development

No branches or pull requests

5 participants