Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

text/template: make and/or operators short-circuit evaluation #31103

Closed
bep opened this issue Mar 28, 2019 · 22 comments
Closed

text/template: make and/or operators short-circuit evaluation #31103

bep opened this issue Mar 28, 2019 · 22 comments
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge Proposal Proposal-Accepted
Milestone

Comments

@bep
Copy link
Contributor

bep commented Mar 28, 2019

Note that the documentation is correct regarding this, so I'm not claiming this is a bug/regression etc. But I suspect the original restriction was motivated by "simpler to implement" -- and since this has become increasingly painful for me lately, I'm testing the water with this issue.

There are at least 3 good arguments in favor of my case:

  1. To avoid nil-pointer situations.
  2. To avoid expensive data initialization when not needed.
  3. The behavior would match the most common boolean AND and OR operators in Go. The current situation is very surprising to most people.
package main

import (
	"io/ioutil"
	"log"
	"text/template"
)

type I interface {
	B() bool
}

type T struct {
}

func (t *T) B() bool {
	return true
}

func main() {
	var t I
	templ, _ := template.New("").Parse("{{ if and .Foo .Foo.B }}TRUE{{ end }}")
	if err := templ.Execute(ioutil.Discard, map[string]interface{}{"Foo": t}); err != nil {
		log.Fatal(err)
	}

}

https://play.golang.org/p/gGdMuJ3-3er

Prints:

2019/03/28 08:45:28 template: :1:19: executing "" at <.Foo.B>: nil pointer evaluating interface {}.B
@robpike
Copy link
Contributor

robpike commented Mar 28, 2019

The execution engine really isn't set up to do this. It's not the answer you want, but use a nested if to avoid unwanted evaluation.

@bep
Copy link
Contributor Author

bep commented Mar 28, 2019

The execution engine really isn't set up to do this. It's not the answer you want, but use a nested if to avoid unwanted evaluation.

I have read and understood the code in text/template, and even if it's "not set up to do this", it should not be too hard to "set it up" to do exactly that. And note that I didn't create this issue only to make the "template life" easier for myself, but I'm on the receiving end of issues/questions from tens of thousands of Go template users, which I think you need to consider before brushing this off as a non-important issue. Because it is.

@robpike
Copy link
Contributor

robpike commented Mar 28, 2019

Please, I am not "brushing this off as a non-important issue". I am saying that the language evaluates all its arguments before calling a function and, even if it's easy to change (I don't think it is), it is unwise to make such a significant change to the language. The documentation does not offer the feature of early out for binary boolean operators, so it's not a bug. And as I wrote, although it's more inconvenient it's not difficult to solve the underlying issue by rewriting the template.

I believe the regularity of the current model is worth preserving.

For the actual example you posted, it's possible there's an unrelated bug. People have been fiddling with nil evaluation in the template engine and it may be that evaluation of methods with nil receivers has been broken, or never worked.

@beoran
Copy link

beoran commented Mar 29, 2019

This would be a backwards incompatible change.
Rather, why not introduce some syntactic sugar where {{ if .Foo && if .Foo.B }}TRUE{{ end }} gets expanded to {{ if .Foo }}{{ if .Foo.B }}TRUE{{ end }}{{ end }} ? That would make nil easier to handle and require no changes to the execution model.

@andybons andybons changed the title text/template: make and/or work like &&/|| proposal: text/template: make and/or work like &&/|| Apr 1, 2019
@gopherbot gopherbot added this to the Proposal milestone Apr 1, 2019
@andybons
Copy link
Member

andybons commented Apr 1, 2019

@mvdan also since improvements have been made to error messages relating to this: #29137

@mvdan
Copy link
Member

mvdan commented Apr 2, 2019

To avoid nil-pointer situations.

Do users misunderstand the nil pointer error and thus file issues on Hugo? I'm asking because perhaps making the errors a bit clearer could avoid confusion amongst users, without the need to make backwards incompatible changes.

To avoid expensive data initialization when not needed.

A template always evaluates all arguments at once, so I don't think this should surprise any user with some experience with the template package. Perhaps it's surprising at first, but we can't change this without breaking backwards compatibility or adding new syntax (see below).

The behavior would match the most common boolean AND and OR operators in Go.

I think you answered your own question here; and and or are functions, not operators with their own syntax that change how the arguments are treated. We have a few scenarios:

  1. Changing all calls to evaluate arguments lazily. I think this would break far too many templates.
  2. Changing the and/or functions to evaluate arguments lazily. This would make these calls inconsistent with others, which would add confusion and break some templates, I think.
  3. Start treating and/or as template language "keywords", as suggested by @bep above. Similar to option 2, but it would still be a breaking change.
  4. Introduce new keywords or syntax, like @beoran's idea above. This is always an option, but it seems overkill given that we can already nest ifs.

I don't think any of these four scenarios is good even in the long run, so I lean towards smaller changes like improving the quality of errors.

@mvdan
Copy link
Member

mvdan commented Apr 2, 2019

For the actual example you posted, it's possible there's an unrelated bug. People have been fiddling with nil evaluation in the template engine and it may be that evaluation of methods with nil receivers has been broken, or never worked.

By the way, I tried the original code on Go 1.11, and it still failed. The nil pointer error did indeed change a bit, but it's largely still the same. If anyone finds a regression in a template that now errors when it didn't on older Go versions, please provide details.

@beoran
Copy link

beoran commented Apr 4, 2019

@mvdan I wouldn't call it overkill when there are more than two conditions. Just compare {{ if .Foo && if .Foo.B && if .Foo.B.C }}TRUE{{ end }} with {{ if .Foo }}{{ if .Foo.B }}{{ if .Foo.C }}TRUE{{ end }}{{ end }}{{ end }}. I'd say the former is just as clear as the latter, but more concise.

@rsc
Copy link
Contributor

rsc commented Apr 10, 2019

I would like to understand better (1) how much work it would be to change them, and (2) whether this would be a breaking change for real templates. Re (2) @beoran asserted that it would, but I can't see why.

@beoran
Copy link

beoran commented Apr 11, 2019

Allow me to give evidence for my 'assertion'.

Now, it is possible to call functions and methods in templates, also as as arguments to the and() function. If these functions or methods have side effects, then as it stands, the side effects of both functions will be performed. If we were to make and() shortcut evaluation, then in case the first function or method returns true, the second function's side effect will not be perfomed anymore. This breaks backwards compatibility as nothing in the documentation of text/template states that methods or functions used with it may not have side effects.

See this for an example:
https://play.golang.org/p/Zwkc1hIhBj5

package main

import (
	"fmt"
	"log"
	"text/template"
	"os"
)

type Data struct {
}

// Method with side effects
func (foo Data) Foo() bool {
	fmt.Println("Foo!")
	return false
}

// Another method with side effects
func (foo Data) Bar() bool {
	fmt.Println("Bar!")
	return false
}

// Simulate a shortcut and function
func Sand(object interface{}) bool {
  	data := object.(Data)
	return data.Foo() && data.Bar()
}

const TEMPLATE = `{{if and (.Foo) (.Bar)}}{{end}}{{if sand .}}{{end}}`

func main() {
	var data Data
	
	funcMap := template.FuncMap{
		// shortcut and
		"sand": Sand,
	}


	tmpl, err := template.New("andTest").Funcs(funcMap).Parse(TEMPLATE)
	if err != nil {
		log.Fatalf("parsing: %s", err)
	}

	err = tmpl.Execute(os.Stdout, data)
	if err != nil {
		log.Fatalf("execution: %s", err)
	}
}

Note the difference between the side effects of the template with and and the one with sand to see why they are not the same.

@bep
Copy link
Contributor Author

bep commented Apr 11, 2019

So, @beoran is right. Kind of. With the changes proposed in this issue, this very unusual template construct might be considered to be broken by some people, fixed by many others.

{{ if (or .InitA .InitB) }}
{{ end }}

I think of this as a proposal with a net positive effect in this area: You may introduce 3 issues in templates, but you will fix 997 already broken template constructs.

@beoran
Copy link

beoran commented Apr 11, 2019

Well, I don't know about the prevalence, I tried to search the web, but searching for {{if (or and {{if (and didn't yield much usable results. I am simply looking at it from a strict go1 compatibility promise point of view. Currently, the behavior as is specified. If the specs need to change, in such a backwards incompatible way, then I consider that this becomes a Go2 issue.

@bep
Copy link
Contributor Author

bep commented Apr 11, 2019

If the specs need to change, in such a backwards incompatible way, then I consider that this becomes a Go2 issue.

Or make this new (sensible) behaviour optional, which would make everyone happy.

@andybons
Copy link
Member

Or make this new (sensible) behaviour optional, which would make everyone happy.

What would you envision the API looking like for this to be opt-in behavior?

@bep
Copy link
Contributor Author

bep commented Apr 12, 2019

What would you envision the API looking like for this to be opt-in behavior?

There is already an options API in place.

tmpl, _ := New("t1").Parse("{{ if (or .InitA .InitB) }}{{ end }}")
tmpl.Option("logicaloperators=legacy/sensible") // or something

@mvdan
Copy link
Member

mvdan commented Apr 13, 2019

Perhaps I'm missing something, but how would that work? You're passing the option after the parsing has completed. I don't think we want to add any form of lazy or delayed parsing to the package.

Edit: I re-read the thread; I assume this option would be to change the evaluation of arguments, not the syntax parsing.

@josharian
Copy link
Contributor

One way to provide options to be used by the parser is to put directives in the template itself. Not sure that we want that, just adding it as a possibility.

@rsc
Copy link
Contributor

rsc commented Apr 24, 2019

I spoke to Rob about this, and we are inclined to try changing the default to short-circuit or/and. It should fix far more than it breaks. Let's try this at the start of the Go 1.14 cycle, and if we need to add an option to get the old non-short-circuit operators back, we can do that.

@rsc rsc modified the milestones: Proposal, Go1.14 Apr 24, 2019
@rsc rsc added the early-in-cycle A change that should be done early in the 3 month dev cycle. label Apr 24, 2019
@rsc rsc changed the title proposal: text/template: make and/or work like &&/|| text/template: make and/or operators short-circuit evaluation Apr 24, 2019
@rsc rsc modified the milestones: Go1.14, Backlog Oct 9, 2019
stephenemslie added a commit to stephenemslie/stripe-ctf-2.0 that referenced this issue Apr 7, 2020
Go templating evaluates all arguments to a function. As a result, and/or
operators will fail under certain conditions when Level is a string rather than a Level struct. Having `Level` potentially be a non Level type is probably bad Go practice anyway. Keeping this type consistent somewhat simplifies the logic of the nav.

This issue with operator evaluation is described in depth here, and
appears to be something that will change:

golang/go#31103
@gopherbot
Copy link

Change https://golang.org/cl/321490 mentions this issue: text/template: implement short-circuit and, or

@gopherbot
Copy link

Change https://golang.org/cl/353130 mentions this issue: text/template: check final value for short-circuit and/or

gopherbot pushed a commit that referenced this issue Sep 29, 2021
There was a bug in the short-circuit code for and/or added in CL 321490:
it ignored the value passed in by an earlier pipeline.

For #31103

Change-Id: Ic31f4d7cedfe563ef968cbb712ecfb2413c42eb5
Reviewed-on: https://go-review.googlesource.com/c/go/+/353130
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
TryBot-Result: Go Bot <gobot@golang.org>
@gopherbot
Copy link

Change https://golang.org/cl/353610 mentions this issue: text/template: undo reflect.Value wrapping for short-circuit and/or

gopherbot pushed a commit that referenced this issue Oct 1, 2021
For #31103

Change-Id: I9c0aa64f95f564de31a4c178e3930584d41316bb
Reviewed-on: https://go-review.googlesource.com/c/go/+/353610
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
TryBot-Result: Go Bot <gobot@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@gopherbot
Copy link

Change https://golang.org/cl/354091 mentions this issue: text/template: only unwrap final and/or value

gopherbot pushed a commit that referenced this issue Oct 5, 2021
In the last CL I missed the fact that except for the final value the
code already unwraps the argument.

For #31103

Change-Id: Ic9099aeb50c6b3322fc14a90ac8026c1d8cb1698
Reviewed-on: https://go-review.googlesource.com/c/go/+/354091
Trust: Ian Lance Taylor <iant@golang.org>
Run-TryBot: Ian Lance Taylor <iant@golang.org>
Reviewed-by: Bryan C. Mills <bcmills@google.com>
@golang golang locked and limited conversation to collaborators Oct 5, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
early-in-cycle A change that should be done early in the 3 month dev cycle. FrozenDueToAge Proposal Proposal-Accepted
Projects
None yet
Development

Successfully merging a pull request may close this issue.

8 participants