I looked into this a bit; Unfortunately I don't think it can work quite the way you've proposed (at least, not with the current testing architecture). In particular, testing likes to emit everything in a text stream, and the JSON blobs are reconstituted with cmd/test2json from that text stream; it would be really tricky to attach metadata to a particular logging statement.

As an additional wrinkle, encoding/json has a dependency on testing, meaning that testing cannot actually use Go's json package for any encoding :(.

It SHOULD be possible, however, to have something which worked like:

func TestFoo(t *testing.T) {
  t.Log("Foo")
  // outputs: {"action": "output", "output": "foo_test.go:12 Foo\n"}
  t.Meta("requestID", "123")
  // outputs: {"action": "meta", "meta": {"requestID": "123"}}
  t.Log("Something Else")
  // outputs: {"action": "output", "output": "foo_test.go:16 Foo\n"}
}

And it could work by emitting an output like:

=== RUN   TestFoo
    foo_test.go:12: Foo
--- META: TestFoo: requestID: 123
    foo_test.go:16: Something Else
--- PASS: TestFoo (N.NNs)

Where ":" is a forbidden character in the key, and the value is trimmed for whitespace.

I think that this functionality might be "good enough" when parsing the test output JSON; metadata would effectively accumulate for the duration of the test, since the test JSON is effectively scanned top-to-bottom anyway to extract information about a given test.

I can write up a CL that we can poke at, if folks don't hate this :)

Actually just went ahead and made a CL: https://go-review.googlesource.com/c/go/+/357914

Change https://golang.org/cl/357914 mentions this issue: testing: allow structured metadata in test2json

+1 for this feature. Being able to set arbitrary metadata would be a great way of helping to get our go test results into a test management system without having to rely on a third party testing library.

I see the CL is kinda stalled out, I can offer time to help push this forward if anything is needed.

Now that we have slog, I wonder if this proposal should be about adding some kind of slog API to testing.T, which is passed through when using test2json?

cc @aclements @dmitshur as I believe y'all have been looking at structured output with cmd/dist.

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

Good discussion on #59928 to figure out how to hook up slog to testing. If we do that, I think that will take care of the need here.

Actually, #59928 (comment) convinced me of the opposite. Slog output should be action="output" but this information should be action="somethingelse".

If we do keep these separate, then I suggest TB.Data(key string, value any), where value is formatted with%q to keep it on one line. "Metadata" is the wrong word here (as it often is elsewhere). This isn't data about data, it's data about tests. I was also thinking of Info, but that might cause confusion with slog, whose Info method takes a message before its keys and values.

One thing I would very much like to have (but maybe cannot 😄 ) is that if value is JSON, that it would be decoded and incorporated into the test2json output without requiring an additional parsing step.

For example:

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could yield

{"action": "data", "key": "mykey", "json": {"hello": "world"}}
{"action": "data", "key": "myotherkey", "str": "hello world"}

IIUC one of the constraints here which makes this unpleasant is that testing cannot depend on encoding/json, so we can't make t.Data(key, value) pass value through encoding/json... However I think test2json can depend on more things, so, maybe, possibly, it could see and decode these?

The proposal in #43936 (comment) looks more practical to me than #43936 (comment).

Specifically, different behavior in

func TestSomething(t *testing.T) {
  t.Data("mykey", `{"hello": "world"}`)
  t.Data("myotherkey", `hello world`)
}

could be a source of a lot of issues. How would it behave with input like t.Data("mykey", "{hello}")? We should only have one way to expose the information. It seems to me that the two keys (json and str) would complicate consumers of the data.

Yeah I think you're right, given the constraints that "testing" cannot validate if a string is valid json or not. Small errors would lead to confusion in the output.

Though I was thinking that, practically, test2json CAN validate, and the str/json division would show prominently when developing a producer/consumer pair. But really it's not a big deal for a consumer to decode the string as json, and pushing it out of test2json also means less overhead in test2json itself, too.

I think a bigger error would be if you had some consumer which expected a string (in some other encoding), but it just SO HAPPENS to be valid json, and test2json decodes it... that would definitely be annoying.

@riannucci How would LUCI make use of the proposed feature (something like t.WithMetadata)? It seems like it helps individual tests get structured output through to something like LUCI. Is that all you are going for? It would not let the overall execution of a test binary be annotated with JSON metadata.

So the original context of my involvement in this proposal was maybe frivolous, but I think the proposal does generally have merit beyond that.

Originally I was interested in this because I was involved with the goconvey project (a now very-outdated testing library which did a lot of interesting things, as well as a lot of not-great things). One of the things this library did was that it reported metadata about every assertion which passed, and displayed these in a web UI... the WAY that it passes this data out from the test is pretty bad though (it dumps JSON objects to stdout at various points in the test execution with funny text markers and then tries to parse them out again. This basically doesn't work with subtests or with very much test parallelism). A lot of the weirdness here is due to goconvey's age - it was originally written in the Go 1.2ish era¹.

I was thinking about how to improve this metadata output situation though, and I think the general objective of "I want to be able to communicate data, correlated with individual tests, from within the test to a higher level tool sitting outside of go test" is a reasonable one. Reporting passing assertions is not very high value, but reporting metrics (or other statistics) or notating the location of testing artifacts (large log files, perhaps outputs from a multi-process testing scenario), etc. I think would be valid use cases.

The direct consumer of such data in LUCI would be ResultDB's streaming test result system; it has the ability to associate test artifacts and other metadata directly with test cases, archiving them to e.g. BigQuery.

It's possible to emulate this, of course, with specially crafted Log lines... but I would prefer if there was some out-of-band way to communicate (even if under the hood, currently, it's really 'testing' and 'test2json' trying their best to produce/parse stdout). I would rather have the 'communication channels' be something that go test owns rather than some other mechanism.

An alternative to this proposal which I thought of, but don't especially like, which would be to produce a second, independent channel/file/pipe from the test binary which only has metadata. There are a number of downsides to this, though:

• Unless go test implements a flag for this, any DIY solution would suffer from a discovery problem (i.e. "does this particular package support this custom flag?"). Goconvey 'solves' this by scanning the test imports which is both slow and weird. Additionally, a custom flag would render the test output uncacheable, which is unfortunate.
• If go test DOES support a way to do this... it seems to me that supporting it via go test -json is probably better than adding an additional flag/output file.
• It would be difficult to correlate the metadata with the other output (log lines/stdout) from the test, which could be useful, though not essential for the cases I have in mind.
• Another alternative would be to set an environment variable for this side channel, but environment variables have their own issues - this could also behave oddly with the test cache, since the test would have different cached output depending on the environment

(Now that I think of it... https://pkg.go.dev/cmd/go#hdr-Test_packages doesn't mention -json as a cacheable flag - is it?)

It would not let the overall execution of a test binary be annotated with JSON metadata.

I understand this to mean "adding metadata to go test as proposed would only allow a test to add metadata scoped to a single named test, not the overall test binary output", which is fine for the cases I had in mind.

edit: formatting

(Oh, I forgot the other bit that goconvey did; for failing assertions it was able to write them out in a structured way, again so that the web UI had better ability to display them; this included things like outputting diffs between actual/expected values)

I think there are several distinct proposals here, all of which are about getting structured information out of tests in some way, but all of which seem to differ significantly in intent:

1. Attaching structured information to individual log lines, where the log lines themselves continue to be regular text. This is my read of the original post. I'm not clear on whether this information should or should not appear in plain text test output because it really is "metadata".
2. A way to emit structured information interleaved with the test log, which is specifically ordered with respect to the reset of the log lines. This is my read of riannucci's comment. I'm not clear how this differs from structured logging in testing. This type of output seems integral to the test log, and thus should be presented in some way even when not in JSON mode. This also doesn't actually seem like "metadata" to me, since it's not data about data, it's just structured data.
3. A way to attach metadata to a test, not in any way ordered with respect to the test log. I'm not sure anyone is asking for this, but this is the other way I could interpret "allowing Go tests to pass metadata".

I think we need concrete use cases to actually move this discussion forward.

From my read of the original post the proposal could arguably be for category 3. That data may also be in regular log output, but the goal is for some other program to read the data. The data doesn't need to be associated with any particular log line, just the test case. The proposal happened to include it with a log line, but the benefits section seems to highlight the "read it from another program" more than the association with a log line.

The use case I'm familiar with is integration with systems like TestRail. My understanding is that they may have their own identifier for a test case separate from the name. And this "metadata" would be a way to associate test cases with their identifier.

As far as I can tell all the use cases described in the original post and in comments are all in category 3. Some of the comments related to log lines were an attempt to propose a solution, but none of the use cases required association or ordering with existing log lines.

Just chiming in to add another use case to the discussion: We use the Go test framework to run a fairly large set of integration tests across our telephony infrastructure. Hundreds of tests and subtests that take about 30 minutes to run in total. Most of these tests start by initiating one or more calls and then check if our services handle the calls and user actions on those calls correctly. Every once in a while, one of these tests will fail after we've made a change or added a new feature. The cause of the failure can not always be found in the logging of the integration tests. Sometimes, something will have gone wrong somewhere in the SIP path and we have to look at logging in other places of our infrastructure. Instead of having to first also dig through the logging of the integration tests to find the associated Call-ID's to query on and such, it would be nice if the Go test framework had a way of exposing some metadata for each test so that we can nicely present it in our test reports (generated from test2json output).

I'm not sure if the Go test framework is intended to be used in this fashion, but figured I'd explain our use case anyway just in case. I believe the proposed t.WithMetadata would work well for us.

Given that artifact saving has been split off into #71287, I think we're all happy with T.Attr(key, value string).

I'd like the discussion on #71287 to get a little further. I'm concerned that if we release T.Attr in one version and then add artifacts in a later version, that test middleware will build its own artifact mechanism on top of T.Attr and then there will be two ways to save artifacts. I don't think #71287 necessarily has to get resolved, I just want it to get to the point where we understand this relationship between it and T.Attr.

With progress on #71287, I'm wondering about correlation versus collation in this domain. I think there's a notion of well-collated or well-correlated results, such that an accurate understanding of the order of events can be established. Beyond unit tests, more on the integration side, I've certainly made mistakes, or had systems misbehave during cleanup, etc. and having an accurate order of events is an essential of the detective work. I'd like testing to be as helpful as possible here.

I'd describe standard testing output (vanilla or test2json) as well-collated here. It's battle tested; various testing issues have been filed addressing collation, and testing source includes several in comments. Collation issues arise directly (#29811, #40771), when resolving panic, #41355, #41479), or in relation to other writes to Stdout/Stderr (#33419). The chatty components of testing enabling test2json are about collation, and T.Output is paying attention to collation. Maybe #71646 ("print more information about failing tests at the end") is making a case that collation is good and we want more.

I get the impression that some proportion of t.Attr use would be for producing well-correlated artifacts. Notionally, that with well-collated and -correlated standard testing output, well-correlated artifacts provide enough information to understand event order. However, I wonder if there are significant edge cases - along the lines of edge cases examined by collation issues - to reason about and execute on here.

An alternative approach that occurs to me is producing well-collated artifacts, via a tracing(-ish) callback at changes in T state. At the checkpoints, the handler would observe a slice of Attrs populate with (at least) the essential bits from within testing to trace the execution testing is orchestrating.

func TraceEvents(t *T, handler func([]testing.Attr))

A notional example here is that I could maintain an event log of systems-under-test, but well-collated/interleaved with events that testing emits.

@AndrewHarrisSPU , I'm not sure I understand your argument. I'll note that I enumerated the expected use cases for test attributes in #43936 (comment). I think a basic property of attributes is that they are clearly correlated with a test, but are not collated with its other output. That's what log messages are for.

@aclements
Pulling back briefly to #43936 (comment) you identified 3 cases of getting structured data out of tests. I'd like to label case 1 as "logs-as-standard-output", and case 2 as "logs-as-artifacts" (the third seems irrelevant).

It sounds like, decisively, t.Attr is case 1, "logs-as-standard-output" approach for t.Attr and I agree that's a pragmatic solution. GitHub actions or LUCI are well-prepared to deal with this. Thumbs-up on t.Attr, for those use cases.

I'm stuck on case-2 "logs-as-artifacts" uses, and I think there's a solution that's not interleaving structured data into the test log, but interleaving testing events into other logs and artifacts. The API could be small: an event-driven callback on testing events. The solution would require a fair amount of wrangling in testing, as testing isn't quite structured that way, but I think it's plausible. But it's a distraction if case-1 is the direction to go with this issue.

We've got at least two concrete use cases for test attributes:

1. JUnit properties. See https://github.com/testmoapp/junitxml?tab=readme-ov-file#properties-for-suites-and-cases. These are a list of string key/value pairs.
2. A Google-internal system (Sponge) that stores test outputs and artifacts. This, too, stores properties as a list of string key/value pairs.

Both of these are nominally ordered (the representation of properties is a list) but do not appear to treat ordering as significant. Neither assigns any form of time order to properties or associates properties with test output. The properties are an additional set of key/value pairs associated with a test run; nothing more.

(I note that both JUinit and Sponge call these "properties" rather than "attributes". Perhaps we should use the "property" terminology as well?)

(I note that both JUinit and Sponge call these "properties" rather than "attributes". Perhaps we should use the "property" terminology as well?)

We called them "attributes" in slog because the standard library uses "attribute" more than "property." I just did a deep dive on that, and it turns out almost all stdlib uses come from the spec being implemented. All of the following systems use "attribute":

• XML
• HTML
• ASN.1
• Dwarf
• Zip
• X.509
• Windows and Unix syscalls

The only system implemented in the stdlib that uses "property" is Unicode.

I mention this for background. I don't think it bears too much on what we should pick here. We could go with "use what's more common in the stdlib" or "use what the reference system(s) use."

We called them "attributes" in slog because the standard library uses "attribute" more than "property." I just did a deep dive on that, and it turns out almost all stdlib uses come from the spec being implemented. All of the following systems use "attribute":

• XML

This might be a reason why JUnit and Sponge, which output XML, do not call them attributes, because it is not a mechanism to add an XML attribute to an element but to add a new type of XML element.

(A rabbit hole in a bike shed...) I wonder if T.TrackingLabel(key, value string) is an affordance, a metaphor about the use cases enabled by the mechanism described here. The name T.Attr seems like it doesn't follow from the mechanism, testing doesn't have an attribute schema. The external tooling dictates the schema, and the mechanism delivers across semantic domains.

Pulling back briefly to #43936 (comment) you identified 3 cases of getting structured data out of tests. I'd like to label case 1 as "logs-as-standard-output", and case 2 as "logs-as-artifacts" (the third seems irrelevant).

The ensuing discussion made is clear that people did in fact want case 3, "a way to attach metadata to a test, not in any way ordered with respect to the test log" and not 1 or 2. This is what JUnit and Sponge use test properties for.

(I note that both JUinit and Sponge call these "properties" rather than "attributes". Perhaps we should use the "property" terminology as well?)

It doesn't help that "attribute" and "property" are defined in terms of each other, at least in Google's dictionary. 😆

Given that "attribute" is widely used for this concept in Go std, even if it is only because it's widely used for this concept across various standards, let's stick with "attribute".

Have all remaining concerns about this proposal been addressed?

The proposal is to add the following to package testing:

type TB interface {
    ...

    // Attr emits a test attribute associated with this test.
    //
    // The key must not contain whitespace.
    //
    // The meaning of different attribute keys is left up to
    // continuous integration systems and test frameworks.
    //
    // Test attributes are emitted immediately in the test log,
    // but they are intended to be treated as unordered.
    Attr(key, value string)
}

Attributes are emitted in the test log as

=== ATTR  TestName <key> <value>

test2json translates these to

{"Action": "attr", "Test": "TestName", "Key": key, "Value": value}

To clarify, is it intentional that the test2json line would exclude remaining TestEvent fields like "Time", "Package" (in contrast to #71287 (comment) where they're included)?

Sorry, I definitely didn't mean to leave out "Package". I think we should include "Time" as well for consistency with all other test events.

Based on the discussion above, this proposal seems like a likely accept.
— aclements for the proposal review group

The proposal is to add the following to package testing:

type TB interface {
    ...

    // Attr emits a test attribute associated with this test.
    //
    // The key must not contain whitespace.
    //
    // The meaning of different attribute keys is left up to
    // continuous integration systems and test frameworks.
    //
    // Test attributes are emitted immediately in the test log,
    // but they are intended to be treated as unordered.
    Attr(key, value string)
}

Attributes are emitted in the test log as

=== ATTR  TestName <key> <value>

test2json translates these to

{"Time": "...", "Action": "attr", "Package": "package/path", "Test": "TestName", "Key": key, "Value": value}

No change in consensus, so accepted. 🎉
This issue now tracks the work of implementing the proposal.
— aclements for the proposal review group

The proposal is to add the following to package testing:

type TB interface {
    ...

    // Attr emits a test attribute associated with this test.
    //
    // The key must not contain whitespace.
    //
    // The meaning of different attribute keys is left up to
    // continuous integration systems and test frameworks.
    //
    // Test attributes are emitted immediately in the test log,
    // but they are intended to be treated as unordered.
    Attr(key, value string)
}

Attributes are emitted in the test log as

=== ATTR  TestName <key> <value>

test2json translates these to

{"Time": "...", "Action": "attr", "Package": "package/path", "Test": "TestName", "Key": key, "Value": value}

Change https://go.dev/cl/662437 mentions this issue: testing: add Attr

Should the documentation be updated to say that the value cannot contain a CR or LF?

type TB interface {
    ...

    // Attr emits a test attribute associated with this test.
    //
    // The key must not contain whitespace.
    //
    // The value must not contain a newline (CR or LF).
    //
    // The meaning of different attribute keys is left up to
    // continuous integration systems and test frameworks.
    //
    // Test attributes are emitted immediately in the test log,
    // but they are intended to be treated as unordered.
    Attr(key, value string)
}

I guess we don't need to mention what happens when the value contains binary data or terminal escape sequences as we don't do that for other methods like Logf.

CR and LF are whitespace, so I don't think we need to call them out specifically.

CR and LF are whitespace, so I don't think we need to call them out specifically.

The value can contain whitespace, only the key cannot.

Did you mean newline? So it would read as follows:

// The value must not contain a newline.

I agree that should be sufficient.

Does the API allow emitting attributes with the same key but different values? e.g.,

t.Attr("log-file", "path.log")
t.Attr("log-file", "path2.log")

As far as I can tell, the intention of this API is that doing this is a mistake (which I agree with). In the wild, I think this mistake will happen somewhat often. e.g., my above "log-file" attribute may come from a test helper that launches a subprocess and adds the path to the process's logs. The helper didn't consider that some tests will launch multiple instances of the subprocess, and thus have multiple log files.

Does the API detect this case? Given the lack of mention in the doc, I assume not. I assume that the attribute is simply written to the output for each value, immediately when Attr is called.

In that case, I think this is insufficient:

    // Test attributes are emitted immediately in the test log,
    // but they are intended to be treated as unordered.

In practice, CI systems will likely select either the first value in the output, or the last. I suspect that once this is widely used Hyrum's law will prevent us from changing the ordering lest we break lots of tests.

In my opinion, we should do one of:

1. Detect and forbid duplicate keys,
2. Randomize the output order, or
3. Explicitly commit to ordering the output by call order.

If we want to enforce uniqueness amongst attribute keys, should they be unique amongst the set of all attribute keys, or unique amongst the set of (test name, attribute key) combinations? Towards the latter, I think testing does enough to promote unique test names (e.g. the Name method of T, B, and F) that this might be reasonable, but I'm also not sure I'm anticipating all possible complications.

For reference, JUnit XML test properties allow duplicate keys, and the choice of how to interpret duplicates is generally up to the consumer.