I looked into this a bit; Unfortunately I don't think it can work quite the way you've proposed (at least, not with the current testing architecture). In particular, testing likes to emit everything in a text stream, and the JSON blobs are reconstituted with cmd/test2json from that text stream; it would be really tricky to attach metadata to a particular logging statement.

As an additional wrinkle, encoding/json has a dependency on testing, meaning that testing cannot actually use Go's json package for any encoding :(.

It SHOULD be possible, however, to have something which worked like:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.Meta("requestID", "123")
  // outputs: {"action": "meta", "meta": {"requestID": "123"}}
  t.Log("Something Else")
  // outputs: {"action": "output", "output": "foo_test.go:16 Foo\n"}
}

And it could work by emitting an output like:

=== RUN   TestFoo
    foo_test.go:12: Foo
--- META: TestFoo: requestID: 123
    foo_test.go:16: Something Else
--- PASS: TestFoo (N.NNs)

Where ":" is a forbidden character in the key, and the value is trimmed for whitespace.

I think that this functionality might be "good enough" when parsing the test output JSON; metadata would effectively accumulate for the duration of the test, since the test JSON is effectively scanned top-to-bottom anyway to extract information about a given test.

I can write up a CL that we can poke at, if folks don't hate this :)

Actually just went ahead and made a CL: https://go-review.googlesource.com/c/go/+/357914

Change https://golang.org/cl/357914 mentions this issue: testing: allow structured metadata in test2json

+1 for this feature. Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

I see the CL is kinda stalled out, I can offer time to help push this forward if anything is needed.

Now that we have slog, I wonder if this proposal should be about adding some kind of slog API to testing.T, which is passed through when using test2json?

cc @aclements @dmitshur as I believe y'all have been looking at structured output with cmd/dist.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Good discussion on #59928 to figure out how to hook up slog to testing. If we do that, I think that will take care of the need here.

Actually, #59928 (comment) convinced me of the opposite. Slog output should be action="output" but this information should be action="somethingelse".

If we do keep these separate, then I suggest TB.Data(key string, value any), where value is formatted with%q to keep it on one line. "Metadata" is the wrong word here (as it often is elsewhere). This isn't data about data, it's data about tests. I was also thinking of Info, but that might cause confusion with slog, whose Info method takes a message before its keys and values.

One thing I would very much like to have (but maybe cannot 😄 ) is that if value is JSON, that it would be decoded and incorporated into the test2json output without requiring an additional parsing step.

For example:

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could yield

{"action": "data", "key": "mykey", "json": {"hello": "world"}}
{"action": "data", "key": "myotherkey", "str": "hello world"}

IIUC one of the constraints here which makes this unpleasant is that testing cannot depend on encoding/json, so we can't make t.Data(key, value) pass value through encoding/json... However I think test2json can depend on more things, so, maybe, possibly, it could see and decode these?

The proposal in #43936 (comment) looks more practical to me than #43936 (comment).

Specifically, different behavior in

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could be a source of a lot of issues. How would it behave with input like t.Data("mykey", "{hello}")? We should only have one way to expose the information. It seems to me that the two keys (json and str) would complicate consumers of the data.

Yeah I think you're right, given the constraints that "testing" cannot validate if a string is valid json or not. Small errors would lead to confusion in the output.

Though I was thinking that, practically, test2json CAN validate, and the str/json division would show prominently when developing a producer/consumer pair. But really it's not a big deal for a consumer to decode the string as json, and pushing it out of test2json also means less overhead in test2json itself, too.

I think a bigger error would be if you had some consumer which expected a string (in some other encoding), but it just SO HAPPENS to be valid json, and test2json decodes it... that would definitely be annoying.

@riannucci How would LUCI make use of the proposed feature (something like t.WithMetadata)? It seems like it helps individual tests get structured output through to something like LUCI. Is that all you are going for? It would not let the overall execution of a test binary be annotated with JSON metadata.

So the original context of my involvement in this proposal was maybe frivolous, but I think the proposal does generally have merit beyond that.

Originally I was interested in this because I was involved with the goconvey project (a now very-outdated testing library which did a lot of interesting things, as well as a lot of not-great things). One of the things this library did was that it reported metadata about every assertion which passed, and displayed these in a web UI... the WAY that it passes this data out from the test is pretty bad though (it dumps JSON objects to stdout at various points in the test execution with funny text markers and then tries to parse them out again. This basically doesn't work with subtests or with very much test parallelism). A lot of the weirdness here is due to goconvey's age - it was originally written in the Go 1.2ish era¹.

I was thinking about how to improve this metadata output situation though, and I think the general objective of "I want to be able to communicate data, correlated with individual tests, from within the test to a higher level tool sitting outside of go test" is a reasonable one. Reporting passing assertions is not very high value, but reporting metrics (or other statistics) or notating the location of testing artifacts (large log files, perhaps outputs from a multi-process testing scenario), etc. I think would be valid use cases.

The direct consumer of such data in LUCI would be ResultDB's streaming test result system; it has the ability to associate test artifacts and other metadata directly with test cases, archiving them to e.g. BigQuery.

It's possible to emulate this, of course, with specially crafted Log lines... but I would prefer if there was some out-of-band way to communicate (even if under the hood, currently, it's really 'testing' and 'test2json' trying their best to produce/parse stdout). I would rather have the 'communication channels' be something that go test owns rather than some other mechanism.

An alternative to this proposal which I thought of, but don't especially like, which would be to produce a second, independent channel/file/pipe from the test binary which only has metadata. There are a number of downsides to this, though:

• Unless go test implements a flag for this, any DIY solution would suffer from a discovery problem (i.e. "does this particular package support this custom flag?"). Goconvey 'solves' this by scanning the test imports which is both slow and weird. Additionally, a custom flag would render the test output uncacheable, which is unfortunate.
• If go test DOES support a way to do this... it seems to me that supporting it via go test -json is probably better than adding an additional flag/output file.
• It would be difficult to correlate the metadata with the other output (log lines/stdout) from the test, which could be useful, though not essential for the cases I have in mind.
• Another alternative would be to set an environment variable for this side channel, but environment variables have their own issues - this could also behave oddly with the test cache, since the test would have different cached output depending on the environment

(Now that I think of it... https://pkg.go.dev/cmd/go#hdr-Test_packages doesn't mention -json as a cacheable flag - is it?)

It would not let the overall execution of a test binary be annotated with JSON metadata.

I understand this to mean "adding metadata to go test as proposed would only allow a test to add metadata scoped to a single named test, not the overall test binary output", which is fine for the cases I had in mind.

edit: formatting

(Oh, I forgot the other bit that goconvey did; for failing assertions it was able to write them out in a structured way, again so that the web UI had better ability to display them; this included things like outputting diffs between actual/expected values)

I think there are several distinct proposals here, all of which are about getting structured information out of tests in some way, but all of which seem to differ significantly in intent:

1. Attaching structured information to individual log lines, where the log lines themselves continue to be regular text. This is my read of the original post. I'm not clear on whether this information should or should not appear in plain text test output because it really is "metadata".
2. A way to emit structured information interleaved with the test log, which is specifically ordered with respect to the reset of the log lines. This is my read of riannucci's comment. I'm not clear how this differs from structured logging in testing. This type of output seems integral to the test log, and thus should be presented in some way even when not in JSON mode. This also doesn't actually seem like "metadata" to me, since it's not data about data, it's just structured data.
3. A way to attach metadata to a test, not in any way ordered with respect to the test log. I'm not sure anyone is asking for this, but this is the other way I could interpret "allowing Go tests to pass metadata".

I think we need concrete use cases to actually move this discussion forward.

From my read of the original post the proposal could arguably be for category 3. That data may also be in regular log output, but the goal is for some other program to read the data. The data doesn't need to be associated with any particular log line, just the test case. The proposal happened to include it with a log line, but the benefits section seems to highlight the "read it from another program" more than the association with a log line.

The use case I'm familiar with is integration with systems like TestRail. My understanding is that they may have their own identifier for a test case separate from the name. And this "metadata" would be a way to associate test cases with their identifier.

As far as I can tell all the use cases described in the original post and in comments are all in category 3. Some of the comments related to log lines were an attempt to propose a solution, but none of the use cases required association or ordering with existing log lines.

Just chiming in to add another use case to the discussion: We use the Go test framework to run a fairly large set of integration tests across our telephony infrastructure. Hundreds of tests and subtests that take about 30 minutes to run in total. Most of these tests start by initiating one or more calls and then check if our services handle the calls and user actions on those calls correctly. Every once in a while, one of these tests will fail after we've made a change or added a new feature. The cause of the failure can not always be found in the logging of the integration tests. Sometimes, something will have gone wrong somewhere in the SIP path and we have to look at logging in other places of our infrastructure. Instead of having to first also dig through the logging of the integration tests to find the associated Call-ID's to query on and such, it would be nice if the Go test framework had a way of exposing some metadata for each test so that we can nicely present it in our test reports (generated from test2json output).

I'm not sure if the Go test framework is intended to be used in this fashion, but figured I'd explain our use case anyway just in case. I believe the proposed t.WithMetadata would work well for us.

I don't think we understand exactly what we need here yet. slog is attractive because it provides the kind of structured data we're talking about, but it's also probably wrong since we want data associated with the test, not a specific log line (see in particular this example from @dnephin). That suggests that we don't want

t.WithMetadata(map[string]string{"requestID": "123"}).Errorf("Foo failed")

but instead we just want something more like:

t.AddMetadata(map[string]string{"requestID": "123"})

That is, it seems like the metadata should not be attached to a specific error message, just to the test itself.

slog is attractive as a way to write structured data, just not the "logging a message" part.

I wonder if we should reuse slog's attribute syntax though. We could add

t.Attr(key, value string)

and define that the attribute list is exactly as defined by slog, including being allowed to pass slog.Attrs. These would be emitted in an

=== ATTR  TestName <key> <value>

line in the output, and in test2json mode would also appear in a

{"Action": "attr", "Test": "TestName", "Attr": {"Key": key, "Value": value}}

line.

Or we could go whole hog and say t.Attrs takes a ...any that it hands to slog to turn into a record and then marshals the record. In test2json mode slog's JSON would end up in the action as

{"Action": "attr", "Test": "TestName": "Attr": anything}

I'm brainstorming here, not arguing for a specific thing.

@jba, @bcmills, any thoughts on adding t.Attr as in my previous comment?

I think t.Attr(key string, value any) is a good middle ground between your two suggestions. You get only one attribute, but the value is whatever you want. The result would be similar to slog.TextHandler's output on slog.Any(key, value), maybe with some tweaks to void newlines in the result.

1. If a test fails, then the program that's analyzing the failed test's json can receive metadata about why it failed: such as requestID, userID etc and then provide the user helpful links to logs and queries.

That does seem useful, but I'm not sure why that would be associated with a test function rather than a specific log line — that is, I'm back to wondering why we should prefer a testing-specific API for that rather than, say, a slog.Logger that emits some kind of structured output to the testing.T log buffer.

2. A test can provide source-code information about where things failed.

We now have #62728 approved for that independently.

@bcmills, see #59928 (comment). In short, providing a slog.Logger that works like t.Log, which is the subject of #59928, is intended to capture log output for the system under test. This proposal is about the test code itself providing data to some downstream system. In the examples discussed here, that data is associated with the test, not a specific log line.

Here's an example that would actually be useful to me (though rarely). There are tests for pkgsite that load a page in a headless browser, save the resulting image, and diff it with a golden image. When those fail, I have to find the path to the diff file in the test output and load the file into a browser. If that file was a piece of data associated with the test, a tool could do that automatically and perfectly.

Sure, you could use the slog.Logger for that, or even t.Log today, but as the comment I linked above and this issue's top post explain, that is not a great solution.

I wrote key, value string above, but I think I meant

t.Attr(args ...any)

where the args are interpreted as they would be in slog. Do people like that?

t.Attr(args ...any)

I don't like it as much as t.Attr(key string, value any).

where the args are interpreted as they would be in slog

So can I pass a slog.Attr instead of a key and value, as in slog.Log? If so, it seems odd that slog.Attr is making an appearance in testing.

In slog, we exchanged the safety of a stricter API for the convenience of multiple alternating keys and values. Here I don't see the use for multiple attributes. (I barely see the use for one.) We can have a less error-prone API.

t.Attr still seems like an awkward fit to me. The original example is associating two nouns — a test, and the ID of a request issued by the test — by connecting them with a verb (“request {ID: 123} failed”). t.Attr associates the attribute with a specific one of those nouns (the test), but doesn't associate either with the verb that connects them.

For example, one might log separate messages for “request started” and “request failed”, and if the test happens to issue two concurrent requests (maybe it's a test of a concurrency property!), it has no way to associate a given request started message with the ID of that individual request.

An explicit slog.Logger instance, in contrast, would allow the metadata to be associated with a specific log message.

@bcmills the original example you referenced is not a representative example of the discussion that followed. I believe we should ignore it, and I propose we remove it from the description because at this point it seems like it's a distraction.

Let's review what has been discussed so far. First, the original proposal

It would be great if Go Tests can attach metadata to be included in the JSON output of a test2json run.
If a test fails, then the program that's analyzing the failed test's json can receive metadata about why [the test] failed ... and then provide the user helpful links to [external] logs and queries.

The request is for data about a test, not about a particular operation the test performed. I believe we all agree that the second point from the proposal is already covered by #62728, as you've already mentioned.

The proposal also says:

[An alternative would be to] include directives in the output string that the json-parsing program can analyze to see if there's metadata. But this solution is very fragile and prone to error.

This is the first indication that mixing this with logging is not a good solution, but there were many more to follow. First let's look at all the use cases that support the idea of this being data about a test case (not about a log line).

#43936 (comment) says

Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

#43936 (comment) says

I think the general objective of "I want to be able to communicate data, correlated with individual tests, from within the test to a higher level tool sitting outside of go test" is a reasonable one. Reporting passing assertions is not very high value, but reporting metrics (or other statistics) or notating the location of testing artifacts (large log files, perhaps outputs from a multi-process testing scenario), etc.

#43936 (comment) references

The use case I'm familiar with is integration with systems like TestRail. My understanding is that they may have their own identifier for a test case separate from the name. And this "metadata" would be a way to associate test cases with their identifier.

#43936 (comment) says

it would be nice if the Go test framework had a way of exposing some metadata for each test so that we can nicely present it in our test reports

All of these are great examples of use cases where we need to associate data with a test. This suggests the t.Attr approach is the correct solution here.

If we look for use cases for attaching data to logging, I think we'll notice there are none! There is not a single use case that would be better served by attaching the data to a specific log line, and there are many examples of why that would not work well.

#43936 (comment) looked into how easy it would be and found problems because of how test2json needs to decode the text format to encode it into JSON.

#43936 (comment) started a discuss about slog, but without presenting any use cases that required it. It was a thought that was explored, and was found to not be a good fit (1, 2).

#43936 (comment) says it well

see #59928 (comment). In short, providing a slog.Logger that works like t.Log, which is the subject of #59928, is intended to capture log output for the system under test. This proposal is about the test code itself providing data to some downstream system. In the examples discussed here, that data is associated with the test, not a specific log line.

All of this points at t.Attr being a more appropriate solution to the problem than anything associated with logging.

one might log separate messages for “request started” and “request failed”, and if the test happens to issue two concurrent requests (maybe it's a test of a concurrency property!), it has no way to associate a given request started message with the ID of that individual request.

That's fine, the test author can and should log separate messages for request started and request failed. If the test author wants to communicate that data using t.Attr it can do that easily:

t.Attr("request.start.0", ...)
t.Attr("request.start.1", ...)
t.Attr("request.end.0", ...)
...

t.Attr(key string, value any)

Seems more appropriate given this encoding in the go text output format:

=== ATTR  TestName <key> <value>

Including multiple key/value pairs in that line would make parsing it into JSON impractical because value (at least) could contain spaces. Each key/value will need to be on a separate line:

=== ATTR  TestName <key1> <value1>
=== ATTR  TestName <key2> <value2>

That makes t.Attr(key string, value any) more consistent with the existing methods on testing.T. A t.Attr(...any) looks like t.Log(...any), but would output differently (the first prints multiple lines, the second a single line).

A couple question about the implementation:

1. How will test2json handle spaces in keys? Are spaces not allowed in keys?
2. Are there any limitations on the contents of value? Can it include newlines? How is it encoded so that test2json can turn it into JSON?

If we want to support json values in test2json then it seems like value needs to be encoded as JSON in text go test output, otherwise it might not be possible to properly encode it as json in test2json.

I'm quite happy with the proposed t.Attr(key string, value any), but I still wonder if t.Data(any) might be easier to handle in test2json:

t.Data(map[string]any{"key": "testID", "value": "abafaf"})
t.Data(testIDs) // where testIDs is a struct that can be JSON marshalled

With the -json flag to go test, we could write t.Attr(key string, value any) as

=== ATTR  TestName <key> <value>

where <key> is a JSON string and <value> is a JSON value, encoded on a single line.

test2json could easily split that into parts and output

{"Action": "attr", "Test": "TestName", "Key": key, "Value": value}

It wouldn't even need to parse JSON to make that transformation. It just has to know how to skip over a JSON string, so it can find the whitespace between <key> and <value>.

So I agree @dnephin, encoding the value as JSON from the start makes the most sense.

To answer your questions: spaces are allowed in keys, and the restrictions on the value are same as for json.Marshal. In particular, newlines in strings are fine.

If go test is called without -json, we have a choice. We could do the exact same thing, or we could output a more human-readable form, closer to slog's TextHandler. I'd vote for the former, since JSON is readable enough and the main use for this feature is with -json.

I still don't think we understand exactly what we need here yet. I looked at gotestyourself/gotestsum#311 (comment) but I don't understand that either. If the test ID would be coming from an environment variable then it seems like it would be the same for every test, which suggests not something that you do with a testing.T. The link to TestRail goes to a marketing page, not something that explains how TestRail uses IDs.

Can someone explain how they would use this, with links to the systems they would use it with - not just advertising pages but the technical details of how those systems would consume the attributes?

My use case is getting our go test results into Allure (https://github.com/allure-framework). Allure is a test report tool that allows you to filter test results on a variety of metadata as well as display metadata in the report such as links to ticket tracking software and attachments (like an output file for a specific test). A live demo is here. A diagram of the Allure processing model is here: https://allurereport.org/docs/how-it-works/

There is prior art for Go Allure adaptors:

• https://github.com/ozontech/allure-go
• https://github.com/dailymotion/allure-go

However, both of these require writing Allure tests, not Go tests. I'd prefer to be able to just annotate my Go tests with the various metadata fields Allure tracks and then post process my go test json into the Allure result json.

There is prior art for this as well here https://github.com/ilyubin/gotest2allure. This provides a version of this functionality by emitting specially constructed log messages and then using a custom parser to extract the metadata back out

I'd like to make round-tripping this metadata more robust than relying on specific log message prefixes, and it would be nice if I didn't have to tie myself to a specific report framework in my implementation. Ideally data like a link to a ticket tracker should be able to be implemented generically within my test, and provided it is easily discoverable in the json output, any test reporting framework should then be able to pull the link back out with a suitable adaptor.

This proposal is becoming more like a partial solution to #41878

@seankhliao #41878 seems to be about getting access to the data that is already provided by test2json. This proposal is about data that is not yet available in test2json output. I don't see how they overlap. Can you elaborate?

still don't think we understand exactly what we need here yet.

@rsc I think there are two use cases that have been shared so far. I'll try to summarize.

Use case 1 - integration with test management systems

Some test management systems track tests using their own IDs. Anyone using these systems needs a way to map the Go test name to the ID used by the test management system. Existing solutions are unreliable and require parsing log output.

Examples: TestRail, Allure framework

This example is in python, but I think it makes it very clear: https://www.browserstack.com/docs/test-management/upload-reports-cli/frameworks/pytest. Each test uses record_property("id", "TC-#####") to identify itself using the TestID of the test management system.

t.Attr would make this possible in Go. Each test uses t.Attr("testID", "..."), a tool would parse the test2json output, find the testID attribute, build a report from that data, and send the report to the test management systems API.

Use case 2 - links to external logs and reports

Any test that runs multiple goroutines (or multiple processes) that produce significant log output need some place to store the logs. Trying to output all the logs from concurrent operations to a single stream often makes the test output unusable.

Instead of a mess of interleaved logs, each process or goroutine writes the logs to its own log file. When a test fails the user may need to dig through those log files to find more information about the failure.

There's no convenient way to expose those logs files (or links to logs) from the test2json output today.

Examples: integration tests across telephony infrastructure, I've experience this same problem when working with https://github.com/hashicorp/consul test suite.

Ci systems have some basic support for this today using test artifacts (see artifacts docs on github actions, circleci, gitlab CI), but all of those require local files, there's no way to create a link to external files.

t.Attr solves this problem. Each test would use t.Attr("processname.logfile", "https://..."), a tool would parse the test2json output, and create a report of failed tests that provides links to external artifacts. This would make it much easier to debug large integration tests.

#41878 is about giving the user the raw data to output in a format they need directly, rather than trying to parse go test's output. Additional metadata may be a part of that.

Rather than trying to hack links together with attr references, #59928 would be a better solution to output from parallel tests.

I would expect that #41878 still needs some way to become aware of this data. t.Attr seems like that mechanism, so my impression is that #41878 could complement this proposal, but does not replace it.

Rather than trying to hack links together with attr references, #59928 would be a better solution to output from parallel tests.

I am looking forward to #59928, but I don't think it solves this problem either.

In one of the examples I linked the tests exercise a system that spans multiple machines. The logs for those requests are not in the current process, so t.Output() doesn't help. The test code can construct a url to find relevant logs from whatever system receives them, but trying to export all of the logs to the test process is often not feasible.

Even when all the logs are available on the machine, if any of the concurrent processes/goroutines have debug logging enabled, trying to output all of the logs to stdout makes the test difficult to debug. It's much easier when those verbose logs are sent to separate files and only the test writes to stdout.

So @nine9ths, would the functions in pkg/allure change like this?

Before:

func Feature(t *testing.T, feature string) {
	t.Logf("%s%s", prefix.Feature, feature)
}

After:

func Feature(t *testing.T, feature string) {
	t.Attr("feature", feature)
}

@jba presumably, or better yet the entire pkg/allure goes away and I just call t.Attr("feature", feature) directly in my tests.