Skip to content

proposal: testing/synctest: create bubbles with Start rather than Run #73062

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
neild opened this issue Mar 26, 2025 · 7 comments
Open

proposal: testing/synctest: create bubbles with Start rather than Run #73062

neild opened this issue Mar 26, 2025 · 7 comments
Labels
Milestone

Comments

@neild
Copy link
Contributor

neild commented Mar 26, 2025

This is a revised version of #67434. I'm creating it as a separate proposal to make it easier to discuss the proposed changes.

In Go 1.24, we added the testing/synctest package as an experiment, enabled by setting GOEXPERIMENT=synctest. See #67434 and https://go.dev/blog/synctest for more details. So far, response has been reasonable favorable.

A few issues have been consistently raised as pain points in the existing API, notably:

The first concern (long-lived background pollers) may be addressed by advancing time within the bubble only so long as the bubble's ancestor goroutine exists. After it exits, we will stop advancing time. If any goroutines are still alive and sleeping or reading from a ticker, we will report a deadlock and panic. This requires no changes to the existing API.

The second two concerns (T.Cleanup and T.Context) point at a need to make testing.T (and B and F) bubble-aware. There are a variety of possible approaches to doing this, but so far as I can tell they all require changes to the testing/synctest API. This is one of them.

// Package synctest provides support for testing concurrent code.
package synctest

// Start places the current goroutine into an isolated "bubble".
// The current goroutine must not be part of a bubble.
//
// Goroutines inherit the bubble of their creating goroutine.
//
// Goroutines in the bubble use a synthetic time implementation.
// The initial time is midnight UTC 2000-01-01.
//
// Time advances when every goroutine in the bubble is blocked.
// For example, a call to time.Sleep will block until all other
// goroutines are blocked and return after the bubble's clock has
// advanced. See [Wait] for the specific definition of blocked.
//
// Time no longer advances within the bubble after the goroutine
// which called Start exits.
//
// If every goroutine in the bubble is blocked and there are no timers scheduled,
// a fatal panic occurs.
//
// Channels, time.Timers, and time.Tickers created within the bubble
// are associated with it. Operating on a bubbled channel, timer, or ticker
// from outside the bubble panics.
func Start()

// Wait is unchanged from the existing proposal.
func Wait()

This proposal changes the Run function to a Start function.

The Run function has some nice properties that we lose: Passing a function to Run makes the bubble lifetime very clear in a way that Start does not, and Run provides a good place to produce a panic when a bubble deadlocks. (In the revised proposal with Start, a deadlocked bubble is now a fatal panic.)

However, Start has the advantage of allowing us to move the point of bubble cleanup into the testing package. I additionally propose that:

  1. When T.Context is called within a bubble, it returns a context with a Done channel that belongs to the bubble.
  2. When T.Cleanup is called within a bubble, the cleanup function is run inside the bubble.
  3. When a test function returns, if the test called synctest.Start the testing package arranges to wait for goroutines in the bubble to exit before finishing the test.

As an example, converting one of the examples from https://go.dev/blog/synctest:

func TestAfterFunc(t *testing.T) {
    synctest.Start()

    ctx, cancel := context.WithCancel(context.Background())
    funcCalled := false
    context.AfterFunc(ctx, func() {
        funcCalled = true
    })

    cancel()
    synctest.Wait()
    if !funcCalled {
        t.Fatalf("AfterFunc function not called after context is canceled")
    }
}

Migration

The testing/synctest package is experimental, so we are free to make incompatible changes to it. However, it would be good to avoid causing problems for anyone kind enough to try out the package in its current experimental state.

I propose that we add synctest.Start and leave synctest.Run in place for now. If we promote synctest out of its experimental status, the Run function will continue exist only when GOEXPERIMENT=synctest is enabled. If synctest continues to exist in Go 1.26 (as an experiment or otherwise), we will remove the Run function in that version.

Alternatives

These are some alternative APIs that satisfy the same goals as this proposal. I don't have a strong preference for any of them; I chose the above proposal because it keeps the synctest API small (still only two functions) and contained within a single package.

Make bubbles a feature of the testing package:

package testing

// Isolate is equivalent to synctest.Start: It places the current goroutine into a bubble.
func (t *T) Isolate() *Bubble
func (b *B) Isolate() *Bubble
func (f *F) Isolate() *Bubble

// For organizational purposes, we move synctest.Wait to a method of a Bubble type.
// If we find a need to add additional bubble operations in the future,
// this will keep them grouped together in the documentation.
type Bubble
func (b *Bubble) Wait()

Make bubbles a feature of the testing package, but keep the Run function:

package testing

// RunIsolated is synctest.Run, but the T, B, or F has bubble-aware Context and Cleanup methods.
func (t *T) RunIsolated(name string, func(*testing.T, *Bubble))
func (b *B) RunIsolated(name string, func(*testing.B, *Bubble))
func (f *F) RunIsolated(name string, func(*testing.F, *Bubble))

type Bubble
func (b *Bubble) Wait()

Same as one of the above, but make the Wait operation a method of testing.TB:

package testing

// One of these:
func (t *T) Isolate() *Bubble
func (t *T) RunIsolated(name string, func(*testing.T, *Bubble))

// Sync replaces synctest.Wait.
func (t *T) Sync()
func (t *B) Sync()
func (t  *F) Sync()

Keep the synctest package, but allow it to create a bubble-aware testing.T/B/F:

package synctest

// Test is synctest.Run, but it provides the function with a T, B, or F with bubble-aware Context/Cleanup.
func Test[T *testing.T | *testing.B | *testing.F](t T, func(T))
@apparentlymart
Copy link

Thanks for the new proposal! Out of curiosity I tried to rewrite the example you included for each of the alternatives you presented, and I figured I might as well share them here to save others the trouble.

Original, just as written in the current text of the proposal:

func TestAfterFunc(t *testing.T) {
    synctest.Start()

    ctx, cancel := context.WithCancel(context.Background())
    funcCalled := false
    context.AfterFunc(ctx, func() {
        funcCalled = true
    })

    cancel()
    synctest.Wait()
    if !funcCalled {
        t.Fatalf("AfterFunc function not called after context is canceled")
    }
}

Make bubbles a feature of the testing package:

func TestAfterFunc(t *testing.T) {
    bubble := t.Isolate()

    ctx, cancel := context.WithCancel(context.Background())
    funcCalled := false
    context.AfterFunc(ctx, func() {
        funcCalled = true
    })

    cancel()
    bubble.Wait()
    if !funcCalled {
        t.Fatalf("AfterFunc function not called after context is canceled")
    }
}

Make bubbles a feature of the testing package, but keep the Run function:

func TestAfterFunc(t *testing.T) {
    t.RunIsolated("bubbled", func (t *testing.T, bubble *testing.Bubble) {
        ctx, cancel := context.WithCancel(context.Background())
        funcCalled := false
        context.AfterFunc(ctx, func() {
            funcCalled = true
        })

        cancel()
        bubble.Wait()
        if !funcCalled {
            t.Fatalf("AfterFunc function not called after context is canceled")
        }
    })
}

Same as one of the above, but make the Wait operation a method of testing.TB:

func TestAfterFunc(t *testing.T) {
    // Ignoring the return value here because it doesn't seem to
    // actually be useful. Was the intention to change this to not return a
    // value for this variant?
    t.Isolate()

    ctx, cancel := context.WithCancel(context.Background())
    funcCalled := false
    context.AfterFunc(ctx, func() {
        funcCalled = true
    })

    cancel()
    t.Sync()
    if !funcCalled {
        t.Fatalf("AfterFunc function not called after context is canceled")
    }
}
func TestAfterFunc(t *testing.T) {
    // NOTE: I assume the intention was to remove the bubble arg
    // here because otherwise it would be unused in this example.
    t.RunIsolated(func (t *testing.T) {
        ctx, cancel := context.WithCancel(context.Background())
        funcCalled := false
        context.AfterFunc(ctx, func() {
            funcCalled = true
        })

        cancel()
        t.Sync()
        if !funcCalled {
            t.Fatalf("AfterFunc function not called after context is canceled")
        }
    })
}

Keep the synctest package, but allow it to create a bubble-aware testing.T/B/F:

func TestAfterFunc(t *testing.T) {
    synctest.Run(func (t *testing.T) {
        ctx, cancel := context.WithCancel(context.Background())
        funcCalled := false
        context.AfterFunc(ctx, func() {
            funcCalled = true
        })

        cancel()
        synctest.Wait()
        if !funcCalled {
            t.Fatalf("AfterFunc function not called after context is canceled")
        }
    })
}

I assume the context.Background() in each of these examples could be replaced by t.Context() with essentially-equivalent effect, since the automatic cancellation when the test completes would act as a no-op for this example.

@apparentlymart
Copy link

apparentlymart commented Mar 26, 2025

On balance, I think of all of these alternatives I lightly prefer the one you labeled "Make bubbles a feature of the testing package":

  • Using a local variable to represent the bubble, rather than package-level functions, communicates clearly to me that this is something I should be manipulating primarily in the test code, and that it is scoped to a particular test without me needing to make any special effort to "clean up" afterwards.

    (I do understand that cleanup is automatic in all of these cases regardless, but for that knowledge I'm relying on context from the proposal discussion so far, whereas this design makes the situation more explicit in the API itself and therefore I expect it to be easier for newcomers to understand.)

  • It avoids the extra level of rightward drift that several of the other variants get from having most of the test code written inside a closure.

    (For the t.RunIsolated variant in particular, I also struggled to think of a useful "name" for the subtest. It seems a little odd to force creating a subtest in order to create a "bubble", since those two concepts seem separate.)

  • Given that the new API here is relatively small, the cost of adding it to package testing (rather than introducing a new package) seems relatively low, particularly if all of the bubble-manipulation functions are methods of Bubble so that they are grouped together nicely in the docs (which you already said, but I agree!).

This is not a strong opinion, though. I think exactly what you proposed would be okay too.

@neild
Copy link
Contributor Author

neild commented Mar 26, 2025

(For the t.RunIsolated variant in particular, I also struggled to think of a useful "name" for the subtest. It seems a little odd to force creating a subtest in order to create a "bubble", since those two concepts seem separate.)

Yes, this is one of the things I've struggled with on making Run a testing.T method: Either it starts a subtest (but what if you don't want a named subtest?) or it doesn't (but its weird for t.Run to start a subtest and t.RunIsolated not to). Or you have one method that doesn't start a subtest and one that does, but now you've got twice the API surface for not much gain.

On the balance, making the create-a-bubble operation move the current goroutine into a bubble seems to let us have the most orthogonal API.

@gh2o
Copy link

gh2o commented Mar 27, 2025

I like the idea of synctest.Run() (and am a little bit wary of synctest.Start()) because it enforces the invariant that a each goroutine is either inside a bubble or outside of bubbles, and there is no way for a goroutine to jump in/out of a bubble. I find this much easier to reason about, and it feels a bit icky to be able to call a third-party function and end up inside a bubble that affects all of my future execution.

My proposed implementation of T.Isolate intentionally avoids transitioning into a bubble by completely restarting the test in a new goroutine.

// Isolate signals that this test is to be run inside an isolated "bubble".
// The test function will first be run as normal. When this function is called
// for the first time, the test function will exit immediately (via runtime.Goexit()),
// and then the test function will be called again from inside a new bubbled
// goroutine. Once inside the bubbled goroutine, the second call to this function will
// be a no-op.
//
// Since any code placed before the Isolate call will be run twice, this should ideally
// be the first function called within the test.
func (t *T) Isolate()

How an example test would run:

func TestSleep(t *testing.T) {
  fmt.Println("hello")

  t.Isolate()

  a := time.Now()
  time.Sleep(24 * time.Hour)
  b := time.Now()

  if b.Sub(a) != 24 * time.Hour {
    t.Error("sleep not exact")
  }
}
  1. The test starts as normal, outside of any bubbles. t is a normal non-bubble-aware T. "hello" is printed.
  2. t.Isolated() is run. T.Isolated() sets some internal flag, then calls runtime.Goexit() (or panics with a known object) to return from the test function.
  3. The testing framework detects this, sees that the internal flag is set, and run this test again.
  4. The test starts a second time, inside a bubble. t is a bubble-aware T. "hello" is printed again.
  5. t.Isolated() is run. T.Isolated() sees that the internal flag is set, and simply returns.
  6. The test proceeds and finishes quickly in real time.

Curious to your thoughts!

@hannahhoward
Copy link

hannahhoward commented Mar 28, 2025

(For the t.RunIsolated variant in particular, I also struggled to think of a useful "name" for the subtest. It seems a little odd to force creating a subtest in order to create a "bubble", since those two concepts seem separate.)

Yes, this is one of the things I've struggled with on making Run a testing.T method: Either it starts a subtest (but what if you don't want a named subtest?) or it doesn't (but its weird for t.Run to start a subtest and t.RunIsolated not to). Or you have one method that doesn't start a subtest and one that does, but now you've got twice the API surface for not much gain.

On the balance, making the create-a-bubble operation move the current goroutine into a bubble seems to let us have the most orthogonal API.

Naming suggestion to avoid confusion -- instead of RunIsolated -- make the method Isolate -- so it's clear it's not a subtest.

so

func TestAfterFunc(t *testing.T) {
    // NOTE: I assume the intention was to remove the bubble arg
    // here because otherwise it would be unused in this example.
    t.Isolate(func (t *testing.T) {
        ctx, cancel := context.WithCancel(context.Background())
        funcCalled := false
        context.AfterFunc(ctx, func() {
            funcCalled = true
        })

        cancel()
        t.Sync()
        if !funcCalled {
            t.Fatalf("AfterFunc function not called after context is canceled")
        }
    })
}

@AndrewHarrisSPU
Copy link

package synctest

// Test is synctest.Run, but it provides the function with a T, B, or F with bubble-aware Context/Cleanup.
func Test[T *testing.T | *testing.B | *testing.F](t T, f func(T))

I'm tempted to impute fork/exec semantics on this, where t is the parent, and Test arranges f(child) more or less as a sub-test of t. I don't think Start() or t.Isolate() ideas are too tricky to use, but I do think this alternative is a little sharper.

@neild
Copy link
Contributor Author

neild commented Mar 31, 2025

My proposed implementation of T.Isolate intentionally avoids transitioning into a bubble by completely restarting the test in a new goroutine.

I've seen this technique (restart the test after calling a function) used to fairly good effect. It's a quite subtle behavior, however, and the edge cases can be surprising and difficult to debug.

For example:

func Test(t *testing.T) {
  time.Sleep(1 * time.Second)
  t.Isolate()
}

I think under the proposed implementation, this sleeps for one real second, restarts the test, sleeps for one virtual second, and then completes. It's even more confusing if the test starts a new goroutine before calling t.Isolate.

The solution to avoid confusion is to not do that: Always call t.Isolate as the very first action in a test. Perhaps we could even have a vet check to enforce this. The API is still sharp-edged, but we could put some safety guards on it.

Another problem is that the test-restarting approach doesn't compose well. t.Parallel marks the current test for parallel execution. Does it matter if t.Parallel is called before or after t.Isolate?

On the balance, I think moving the current goroutine into a bubble has fewer edge cases than restarting tests midway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

6 participants