Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: explicitly scoped defers #54249

Closed
YairHalberstadt opened this issue Aug 4, 2022 · 12 comments
Closed

proposal: Go 2: explicitly scoped defers #54249

YairHalberstadt opened this issue Aug 4, 2022 · 12 comments
Labels
Milestone

Comments

@YairHalberstadt
Copy link

YairHalberstadt commented Aug 4, 2022

Author background

  • Would you consider yourself a novice, intermediate, or experienced Go programmer? intermediate
  • What other languages do you have experience with? C#, scala, python

Related proposals

  • Has this idea, or one like it, been proposed before? I haven't found any, but sometimes proposals are difficult to find.
  • Does this affect error handling? no
  • Is this about generics? no

Proposal

  • What is the proposed change? allow defer to run deterministically at the end of a given scope
  • Who does this proposal help, and why?

Currently defer statements run at the end of a function. This means that resources can live far longer than they should, especially if defer is called inside a loop, when counter-intuitively the defer statements won't be called till the loop exits. This can even cause a memory leak if it's an infinite loop.

This can also be especially problematic if the resource being closed is a lock, which should be taken for the shortest time possible.

The solution to this is either not to use defer in such cases (which means the resource wont be closed in the case of a panic), or to move out the usage of the resource into it's own function.

The latter is certainly a valid solution, but in practice developers are lazy. Given the effort of extracting a function in order to narrow the scope of a defer statement, they are reasonably likely to just leave it as it is, and let the resource live a little longer than it should, especially if extracting the function means modifying complex control flow.

This language change provides a lightweight, easy to use and intuitive syntax to allow scoping a defer to a particular block. This makes it trivial to ensure a resource only lives as long as it needs to.

  • Please describe as precisely as possible the change to the language.

We allow defer to be scoped to a particular lexical block, indicated by starting the block on the same line as the defer statement (similiarly to if). The defer statement will run as soon as the block is exited.

file, err := os.Open(path)
if err != nil {
    return err
}
defer file.Close() {
   // use file here
}
// do other stuff.
  • What would change in the language spec?

syntax

We allow an optional block after a defer statement

DeferStmt = "defer" Expression [ Block ] .

semantics

If a block is provided the "defer" statement invokes a function whose execution is deferred to the moment execution exits the block, either because execution reached the end of the block, or because the corresponding goroutine is panicking.

The semantics of the block itself are exactly the same as a normal block.

  • Is this change backward compatible? Yes
  • Orthogonality: how does this change interact or overlap with existing features? doesn't have much impact
  • Is the goal of this change a performance improvement? No

Costs

  • Would this change make Go easier or harder to learn, and why? slightly harder, but only very slight
  • What is the cost of this proposal? (Every language change has a cost). low
  • How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected? gofmt. We should also add rules to linters to detect cases where defer could be scoped and isn't.
  • What is the compile time cost? low
  • What is the run time cost? zero - if anything more performant when the defer block is called inside a loop, since there's no need to allocate the defer func on the heap.
  • Can you describe a possible implementation? Same as currrent defer, but executing earlier.
  • Do you have a prototype? (This is not required.) no
@gopherbot gopherbot added this to the Proposal milestone Aug 4, 2022
@tkw1536
Copy link

tkw1536 commented Aug 4, 2022

This can be achieved without control flow modification already.

file, err := os.Open(path)
if err != nil {
    return err
}

var closeOnce sync.Once
defer closeOnce.Do(func() { file.Close() })

// use file here
closeOnce.Do(func() { file.Close() })

// do other stuff.

@YairHalberstadt
Copy link
Author

@tkw1536 certainly that's another approach that currently works.

The aim of this proposal is to make doing the right thing so simple, straightforward and lightweight that everybody does it by default, and we can even add lints which use rules of thumb to push people in the right direction.

The fact that the pattern you suggest isn't used in a very high percentage of defers today is evidence we could do with something better here.

@ianlancetaylor ianlancetaylor added LanguageChange v2 A language change or incompatible library change labels Aug 4, 2022
@rittneje
Copy link

rittneje commented Aug 5, 2022

It should be noted that today you can also use anonymous functions for this. (Admittedly it can be a little awkward with return values, but at least you don't have to define a totally separate helper function.)

func foo(path) error {
    err := func() error {
        file, err := os.Open(path)
        if err != nil {
            return err
        }
        defer file.Close()
       // use file here
    }()
    if err != nil {
        return err
    }

    // more stuff here
}

That said, I can definitely sympathize with the desire for a more convenient syntax. However, one thing to note about this proposal is it will lead to a lot of nesting in case there are several defers.

f1, err := os.Open(...)
if err != nil { ... }
defer f1.Close() {
    f2, err := os.Open(...)
    if err != nil { ... }
    defer f2.Close() {
        ...
    }
}

Also this specific proposal introduces a potential ambiguity in the grammar. The following could either be a normal defer statement followed by a block (as is the case today), or a scoped defer statement (your proposal). You could argue that it is clearly the former (to align with how if is handled), but that seems like a common source of mistakes if developers are not careful.

defer f1.Close()
{
    ...
}

@CAFxX
Copy link
Contributor

CAFxX commented Aug 5, 2022

Not a super fan of requiring a dedicated block to scope the defer (to see why imagine you need to have N files open: it's going to quickly start drifting right).

file1, err := os.Open(path1)
if err != nil {
    return err
}
defer file1.Close() {
	  file2, err := os.Open(path2)
	  if err != nil {
	      return err
	  }
	  defer file2.Close() { 
			file3, err := os.Open(path3)
		    if err != nil {
		        return err
		    }
		    defer file3.Close() {
			    // potentially more levels...
			    // use files here
			}
	  }
}

// do other stuff.

Also, it would force all variables in the defer scope to become inaccessible outside the scope, and this would manifest in a weird asymmetry: e.g. in the example above in // do other stuff you would still be able to access the (closed) file1, but not file2 or file3 (unless, obviously, you define them before entering the first defer scope, and then you don't shadow them by mistake).

Not advocating for a specific alternative, but a shorthand syntax that just allowed users to trigger the execution of a defer immediately (and not executing it when the function returns) would be more desirable (that is basically what the sync.Once approach is). Bonus points from me if the new syntax allowed also a future extension to disarm a defer.

@YairHalberstadt
Copy link
Author

@rittneje @CAFxX thanks for your points. All sensible!

Firstly, I think they don't necessarily kill the proposal.

I don't believe the ambiguity point is correct. This is currently illegal:

if true
{
   // Do something
}

syntax error: unexpected newline, expecting { after if clause

And similarly it would be required for the block to begin on the same line as the defer statement. It might be worth having a linter check for a defer followed by a scope on the next line if it ends up being a common pitfall, but I doubt that.

I think the majority of cases don't have nesting defers, so I'm ok having heavily indented files when they do occur (or using an alternative option in such cases). Similarly, based on my experience with C#'s using, in most cases you don't need a variable to escape the block scope, and when you do it's not too difficult to declare the variable up front, much as people do for if and for statements today.

I agree it is unfortunate that the file would still be in scope after the block exits, but that is something a linter should be able to detect in most situations.

That said I have some alternatives, which if people are interested in I'm happy to open proposals for.

alternative 1 - add scoped defer statement

We add a contextual keyword scoped. A scoped defer runs whenever the surrounding block (implicit or explicit) exits.

This allows doing this:

// Introduce a new block here for purpose of scoping file.
{
  file, err := os.Open(path)
  if err != nil {
    return err
  }
  scoped defer file.Close() 
  // use file here

  // defer statement runs here when scope exits
}
// do other stuff.

alternative 2 - defer statements return an instance of the Deffered interface.

We define an interface Deffered:

type Deffered interface {
  Run()
  Cancel()
}

defer returns an instance of this interface.

This would allow explicitly running the deferred statement early or cancelling it as follows:

file, err := os.Open(path)
if err != nil {
    return err
}
closeFile := defer file.Close()
// use file here
closeFile.Run()
// do other stuff.

If Run() is called the defer statement will not execute when the function exits.

The big disadvantage of this approach is that if the user breaks or continues out of the scope before calling Run the defer statement won't be executed till the function exits.

@seankhliao
Copy link
Member

related (dup?) of #38520 #30130

@DeedleFake
Copy link

DeedleFake commented Aug 5, 2022

To fix the nesting, how about the ability to specify a label for a defer, then use the label to specify a scope? In other words, something like

func Example() err {
files:
  {
    file1, err := os.Open(path1)
    if err != nil {
      return err
    }
    defer files, file1.Close()

    file2, err := os.Open(path2)
    if err != nil {
      return err
    }
    defer files, file2.Close()

    // Do stuff with files.
  } // Deferred stuff runs here or when control flow otherwise leaves the labeled block.

  // The defers above have both run by this point.
}

@ianlancetaylor
Copy link
Contributor

I don't know if I like this idea in general, but I do like @DeedleFake 's suggestion quite a bit more than the original proposal on this issue.

I'm not 100% sure but I don't think we need the comma.

@DeedleFake
Copy link

I'm not 100% sure but I don't think we need the comma.

I didn't have the comma originally, but it felt inconsistent with a multi-argument return. This is a tad different, obviously, but at least at a glance the syntax is pretty similar in that it's a keyword with more than one thing being 'passed' to it.

@CAFxX
Copy link
Contributor

CAFxX commented Aug 6, 2022

I'm fine with the label approach, but I would really prefer if there was a way to refer to the current scope without a label (as i would expect the current scope to be the most commonly used, as is the case for break and continue).

Just to clarify: I'm obviously not suggesting to change the current defer behavior

@ianlancetaylor
Copy link
Contributor

As discussed above, there are already ways to scope a defer statement, such as by wrapping it in a function literal. The syntax suggestions here, while sometimes convenient, aren't enough of an improvement to justify adding additional syntax to the language.

Therefore, this is a likely decline. Leaving open for four weeks for final comments.

@ianlancetaylor
Copy link
Contributor

No further comments.

@ianlancetaylor ianlancetaylor closed this as not planned Won't fix, can't repro, duplicate, stale Nov 2, 2022
@golang golang locked and limited conversation to collaborators Nov 2, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

No branches or pull requests

8 participants