testing: failure messages should be easier to find #47080
Labels
NeedsInvestigation
Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
I've spent a lot of time poring over CI test output trying to find exactly why a given test has failed.
Often the output of a high level test includes info-level logs that may end up very useful to determine
the cause of failure but are just noise when looking for the actual failure message.
There is currently no easy way to distinguish between info-level and error-level messages which makes
it hard to find the actual cause of a given failure. Often I end up editing the test output and excluding
all lines that match some regular expression, ending up with a smaller number of candidate lines which
include the culprit I'm searching for.
This is made harder by the fact that some failure messages are generated by calling
T.Log
and thenT.Fail
so it's not necessarily clear that the logged message represents a failure.Perhaps we could include an extra action type in the JSON test output that represents a call to
T.Fail
orT.FailNow
; for example:Perhaps the line we could also include a stack trace of the failure, making it easier for tooling to point directly to the offending code.
This doesn't help with regular non-JSON test output, but that's perhaps hard to fix without breaking people's expected test messages, though we could add a flag to enable an explicit "test failed here" output line.
The text was updated successfully, but these errors were encountered: