Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: x/sync/errgroup: allow manual management of cancel #54079

Closed
phenpessoa opened this issue Jul 27, 2022 · 9 comments
Closed

proposal: x/sync/errgroup: allow manual management of cancel #54079

phenpessoa opened this issue Jul 27, 2022 · 9 comments

Comments

@phenpessoa
Copy link

phenpessoa commented Jul 27, 2022

I ran into an issue while using errgroup and database transactions.
This is similar to what I tried to do:

func main() {
	db := &sql.DB{}

	errGroup, ctx := errgroup.WithContext(context.Background())
	tx, _ := db.BeginTx(ctx, &sql.TxOptions{})

	for i := 0; i < x; i++ {
		errGroup.Go(func() error {
			_, err := tx.Exec("")
			return err
		})
	}

	if err := errGroup.Wait(); err != nil {
		tx.Rollback()
		return
	}

	tx.Commit()
}

This code works just fine in case any Exec returns an error. But it will never work if all Exec calls succeed, reason being tx.Commit will always fail because ctx is canceled by errGroup.Wait.

However, if Wait did not cancel ctx, this code would work as expected.

My proposal is the follow: add two more methods to errgroup.Group: Cancel and WaitNoCancel (names are just a suggestion and can be changed).

This is how they'd look like:

func (g *Group) WaitNoCancel() error {
	g.wg.Wait()
	return g.err
}

func (g *Group) Cancel() {
	if g.cancel != nil {
		g.cancel()
	}
}

This way, the code I wanted to write would look something like this:

func main() {
	db := &sql.DB{}

	errGroup, ctx := WithContext(context.Background())
	tx, _ := db.BeginTx(ctx, &sql.TxOptions{})

	for i := 0; i < x; i++ {
		errGroup.Go(func() error {
			_, err := tx.Exec("")
			return err
		})
	}

	defer errGroup.Cancel()

	if err := errGroup.WaitNoCancel(); err != nil {
		tx.Rollback()
		return
	}

	tx.Commit()
}

This time, tx.Commit would work just fine, because it's ctx would not be canceled by errGroup.WaitNoCancel.

With the current implementation of errgroup, the only way to make this work is to make a Group with no cancel, or to use a different ctx in the transaction. But I don't think this is ideal because Group.Go does not check if the ctx was canceled before calling f, thus all queries/execs in the transaction would be executed, even if the Group has already failed.
Passing the Group's ctx to the transaction prevents this from happening.

@gopherbot gopherbot added this to the Proposal milestone Jul 27, 2022
@ianlancetaylor
Copy link
Contributor

CC @bcmills

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Jul 27, 2022
@rsc
Copy link
Contributor

rsc commented Jul 27, 2022

The errgroup context is provided as a convenience. If it's not convenient, isn't it possible to create the context separate from the errgroup? It does't seem like errgroup needs to change.

That is, use context.WithCancel and then call that cancel func instead of errgroup.Cancel in the rewrite.

@rsc rsc moved this from Incoming to Active in Proposals (old) Jul 27, 2022
@rsc
Copy link
Contributor

rsc commented Jul 27, 2022

This proposal has been added to the active column of the proposals project
and will now be reviewed at the weekly proposal review meetings.
— rsc for the proposal review group

@bcmills
Copy link
Contributor

bcmills commented Jul 27, 2022

I don't think this is ideal because Group.Go does not check if the ctx was canceled before calling f, thus all queries/execs in the transaction would be executed

That is true even with errgroup.WithContext today — so I don't see that this proposal would make that approach any better.

@phenpessoa
Copy link
Author

The errgroup context is provided as a convenience. If it's not convenient, isn't it possible to create the context separate from the errgroup? It does't seem like errgroup needs to change.

That is, use context.WithCancel and then call that cancel func instead of errgroup.Cancel in the rewrite.

You are right, it is possible indeed.

That is true even with errgroup.WithContext today — so I don't see that this proposal would make that approach any better.

That's true. I'd like to rephrase what I said in the original post and give another example to better illustrate my point.

My original post and the title was a bit misleading as to what I wanted to say, I hope to make it clearer this time. Sorry.


While it is true that one could use a different context, I don't think it is a better solution than my proposed one.
More so if the funcs called by the Go method can't be grouped inside a loop.

For example, currently, this is how this could be achieved, as proposed by @rsc, if I understood it correctly:

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

errGroup := errgroup.Group{}
tx, _ := db.BeginTx(ctx, &sql.TxOptions{})

errGroup.Go(func() error {
	var err error
	defer func() {
		if err != nil {
			cancel()
		}
	}()

	_, err = tx.ExecContext(ctx, "exec one")
	return err
})

errGroup.Go(func() error {
	var err error
	defer func() {
		if err != nil {
			cancel()
		}
	}()

	_, err = tx.ExecContext(ctx, "exec two")
	return err
})

errGroup.Go(func() error {
	var err error
	defer func() {
		if err != nil {
			cancel()
		}
	}()

	_, err = tx.ExecContext(ctx, "exec three")
	return err
})

if err := errGroup.Wait(); err != nil {
	tx.Rollback()
	return
}

tx.Commit()

The context would only be canceled in case of an error, because the errorgroup is not using it, and the commit would work just fine.
We are also avoiding unnecessary database calls, because the context is being passed to the exec calls.

But, imo, this looks too repetitive.
With my proposed change, it would look like this:

errGroup, ctx := errgroup.WithContext(context.Background())
defer errGroup.Cancel()
tx, _ := db.BeginTx(ctx, &sql.TxOptions{})

errGroup.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec one")
	return err
})

errGroup.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec two")
	return err
})

errGroup.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec three")
	return err
})

if err := errGroup.WaitNoCancel(); err != nil {
	tx.Rollback()
	return
}

tx.Commit()

This is cleaner and more convinient, imo.

The method Go cancels the context if f errors out. So the user doesn't have to worry about cancelig the context. This avoid code repetition.

And it doesn't matter that the method Go doesn't check the ctx before calling f, because in this case the ctx is passed to the transaction and the exec calls.
In casy any f fails, all subsequent execs would give a ctx error instead of executing the call. And this would have been handled by the errgroup package, instead of manually and repetitively by the user.

Up to this point, this is all doable with the current implementation of the errgroup package.
The trick part here is the WaitNoCancel method. Without it, the commit would always fail, because calling Wait instead would guarantee that the context would always be canceled, even if no error occurred.

@bcmills
Copy link
Contributor

bcmills commented Jul 27, 2022

What is the difference between your proposed approach and this one?

txCtx, cancel := context.WithCancel(context.Background())
defer cancel()
tx, _ := db.BeginTx(txCtx, &sql.TxOptions{})

g, ctx := errgroup.WithContext(txCtx)

g.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec one")
	return err
})

g.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec two")
	return err
})

g.Go(func() error {
	_, err := tx.ExecContext(ctx, "exec three")
	return err
})

if err := g.Wait(); err != nil {
	tx.Rollback()
	return
}

tx.Commit()

@phenpessoa
Copy link
Author

What is the difference between your proposed approach and this one?

I didn't think of that approach. That's a good point.
Guess the only difference is dealing with a single context, instead of two. But I don't think it's enough to justify the change. Nor do I think it's a bad thing to be honest.

Thanks for the feedback!

@rsc
Copy link
Contributor

rsc commented Aug 3, 2022

Based on the discussion above, this proposal seems like a likely decline.
— rsc for the proposal review group

@rsc rsc moved this from Active to Likely Decline in Proposals (old) Aug 3, 2022
@rsc
Copy link
Contributor

rsc commented Aug 12, 2022

No change in consensus, so declined.
— rsc for the proposal review group

@rsc rsc closed this as completed Aug 12, 2022
@golang golang locked and limited conversation to collaborators Aug 12, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Proposals (old)
Likely Decline
Development

No branches or pull requests

5 participants