Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: struct field methods, or anonymous callback helper types #56647

Closed
bford opened this issue Nov 8, 2022 · 6 comments
Closed
Labels
FrozenDueToAge LanguageChange Suggested changes to the Go language Proposal Proposal-FinalCommentPeriod v2 An incompatible library change
Milestone

Comments

@bford
Copy link
Contributor

bford commented Nov 8, 2022

While implementing the main functionality of a Go struct, we often need to define associated "helper" types whose purpose is merely to implement a certain interface with methods that often call right back into the main struct (or directly access the state in the main struct). It would be nice if such "callback helper" types could be anonymous, avoid diverting the developer's attention away from the main struct, and avoid needing explicit back pointers to the main struct. This goal could be accomplished by allowing method declarations attached to fields within structs, as discussed below.

For example, suppose the main struct needs to invoke some standard encoding API like json.NewEncoder() but needs to pass an io.Writer to it. The main struct does not want to be an io.Writer itself: that is, it does not want a Write() method in its own public API. Instead, the main struct just needs a some subsidiary helper object to pass to json.NewEncoder(), which implements the io.Writer interface, and whose custom Write() method calls back into the main struct to be implemented there with access to all the state in the main struct.

Concretely, I find that more-or-less the following pattern occurs often in Go code:

// the main object we want: all the state is here and all the real action happens here
type MainObject struct {
   writer mainObjectWriterCallback  // embedded helper to implement a custom io.Writer
}

func (m *MainObject) Init() {
   writer.backPointer = m // make sure the helper can get back to the MainObject's state
}

func (m *MainObject) EncodeThyself() {
   someCodec.EncodeTo(&m.writer)   // EncodeTo() wants an io.Writer
}

func (m *MainObject) encodeWrite(p []byte) (n int, err error) {
   // implement the Write method that someCodec.EncodeTo() wants, using MainObject's state
}


// the helper type, which exists only to be an io.Writer and forward Write() back to MainObject
type mainObjectWriterCallback struct { 
   backPointer *MainObject 
}

func (mowc mainObjectWriterCallback) Write(p []byte) (n int, err error) {
   n, err = mowc.backPointer.encodeWrite(p)
   return
}

Of course mainObjectWriterCallback.Write() could simply "do the work itself" by following its mowc.backPointer back to the MainObject. But this organization breaks the flow of the logic that really belongs back in MainObject, not in the mainObjectWriterCallback helper. And the separate named type mainObjectWriterCallback is still needed, with its explicit back pointer mowc.backPointer to the main object, which needs to be initialized explicitly in Init(), etc.

Simpler, and more concise, anonymous helper types

I propose that a future version of Go allow such helper type(s) and associated callback forwarding methods to be replaced with code like the following, in which method declarations attach can attach a method directly to a field within a struct:

type MainObject struct { // all the state is here and all the real action happens here
   writer struct{}  // embedded helper: no longer needs a type name, may be empty
}

func (m *MainObject) EncodeThyself() {  // this is a normal struct method
   someCodec.EncodeTo(&m.writer)   // EncodeTo() wants an io.Writer
}

func (m *MainObject) writer.Write(p []byte) (n int, err error) {  // this is a struct field method
   // implement the Write method that someCodec.EncodeTo() wants, using MainObject's state
}

Note that the writer field is declared as an empty, anonymous struct{} in this example. However, attaching method(s) to it like writer.Write() effectively creates an anonymous (sub-)struct type associated with that field, with the associated methods, and which is capable of satisfying any interfaces that might require those methods. Thus, the call to EncodeTo(&m.writer) works because the writer field has a suitable Write() method, even though the field's declared type of struct{} does not. Further, this writer.Write() method implicitly "knows" that it is part of MainObject, not just an associated helper type, and can directly access all of the main struct's state via the primary m pointer, without needing to follow any explicit back pointer from a separate helper type.

Extending existing types with additional methods in anonymous helper types

The declared type need not always be an empty struct{} as in the above example, of course. Instead, the developer can pick any "base" type T and implicitly extend T with additional methods. In the following variation of the above example, MainObject wants to implement writer as a standard bytes.Buffer object, but needs to extend the helper object with an additional WriteSpecialThing() method that someCodec.EncodeTo() demands:

type MainObject struct { // all the state is here and all the real action happens here
   writer bytes.Buffer  // embedded helper type built on the standard bytes.Buffer struct type
}

func (m *MainObject) EncodeThyself() {  // this is a normal struct method
   someCodec.EncodeTo(&m.writer)   // EncodeTo() demands both Write() and WriteSpecialThing() methods
}

func (m *MainObject) writer.WriteSpecialThing(the Thing) error {  // this is a struct field method
   // code to write a Thing, potentially using any state in MainObject, not just in MainObject.writer
}

Implementing field methods and anonymous helper types

A straightforward way to implement this feature would be by automatically rewriting the appropriate field(s) in the main method to have a hidden struct type, which uses the existing struct composition feature to "inherit" the field's declared type with its existing methods but then enhance it with additional methods. For example, the above example in which MainObject.writer extends bytes.Buffer would be functionally equivalent to the following code:

type MainObject struct {
   writer hiddenHelper  // hiddenHelper extends the declared bytes.Buffer type with additional methods
}

type hiddenHelper struct {  // hidden helper struct automatically generated by the Go compiler
   bytes.Buffer   // basic declared type of MainObject.writer to be extended with additional methods
   backPointer *MainObject  // automatically-initialized back pointer to the main object
}
func (hh hiddenHelper) Write(p []byte) (n int, err error) { ... }

A smarter implementation could probably eliminate the need for hidden back pointers at all. Suppose the Go compiler knows that this hidden helper type is only ever instantiated within a MainObject struct (since it's a field of MainObject), and that field is always located at constant offset o within the MainObject struct. Then the implementation of writer.Write - pointed to by the hidden type's method dispatch table - can simply take the pointer it receives to MainObject.writer and subtract offset o to recover the MainObject pointer that the code in that method needs. (This hidden pointer arithmetic happens to be equivalent to what C++ compilers already do to handle multiple inheritance efficiently, but this proposal is not about multiple inheritance.)

Thus, besides the convenience and conciseness benefits of field methods, there are also potential (slight) storage space and efficiency benefits in not having to maintain (or follow) back pointers from helper types associated with a struct type.

Author background

  • Would you consider yourself a novice, intermediate, or experienced Go programmer? Experienced
  • What other languages do you have experience with? C, C++, Java, Haskell, JavaScript, assembly, ...

Related proposals

  • Has this idea, or one like it, been proposed before? Not to my knowledge.
  • Does this affect error handling? No.
  • Is this about generics? No.

Proposal

  • What is the proposed change? See above.
  • Who does this proposal help, and why? Anyone implementing a struct that needs helper types just to implement callback interfaces that some external API demands.
  • Please describe as precisely as possible the change to the language. See above.
  • What would change in the language spec? Method declarations could declare methods of fields, like func (m *MainObject) writer.Write(), whereas traditionally methods can be declared only associated with the "top-level" struct being defined.
  • Please also describe the change informally, as in a class teaching Go. See above.
  • Is this change backward compatible? Yes, except possibly with respect to reflection, which might start "seeing" something new and unexpected when reflecting on a struct. See above for before and after equivalents.
  • Orthogonality: how does this change interact or overlap with existing features? The proposed feature leverages and conceptually builds on the existing struct composition feature already allowing an arbitrary existing type to be extended with new methods. It arguably makes struct composition easier and more useful in an important common case.
  • Is the goal of this change a performance improvement? No. (Although it might eventually enable a slight performance and storage-efficiency improvement depending on how it is implemented, as discussed above.)

Costs

  • Would this change make Go easier or harder to learn, and why? The obvious cost is slightly complicating the language a bit further: having the feature is more complicated than not having the feature. Users will not necessarily need to learn or use it, but my impression is that this design pattern is common enough that users will learn and want to use it.
  • What is the cost of this proposal? (Every language change has a cost). Primarily, a slight complexity increase in both the language and the compiler implementation. Also, potential confusion and subtleties related to the fact that a struct field can now effectively have "two types" at once: the declared field type, and the hidden helper type extended with the additional methods associated with the field.
  • How many tools (such as vet, gopls, gofmt, goimports, etc.) would be affected? All the tools that know about method declarations and would need to be updated to know about field method declarations in order to make sense of them.
  • What is the compile time cost? A slight cost of a slight automatic transformation; should be negligible.
  • What is the run time cost? The runtime cost should be zero or negative (a slight performance improvement) if the feature is implemented sensibly.
  • Can you describe a possible implementation? See above.
  • Do you have a prototype? (This is not required.) No. Only the idea for now.
@gopherbot gopherbot added this to the Proposal milestone Nov 8, 2022
@seankhliao seankhliao added LanguageChange Suggested changes to the Go language v2 An incompatible library change labels Nov 8, 2022
@seankhliao seankhliao changed the title proposal: struct field methods, or anonymous callback helper types proposal: Go 2: struct field methods, or anonymous callback helper types Nov 8, 2022
@zephyrtronium
Copy link
Contributor

Concretely, I find that more-or-less the following pattern occurs often in Go code: ...

Do you have any real-world examples of this pattern? I've never seen it, so it's difficult for me to imagine where this helps.

I do think a slightly different formalism would clarify this proposal. When you say that func (m *MainObject) writer.Write(...) creates a hidden anonymous type for the writer field, and that field then has "two types at once," that is hard to reconcile with my understanding of programming languages in general and Go in particular. If we were to instead say that it creates a method of *MainObject that is scoped to the writer field, that seems to accomplish your goal, and I find it easier to understand.

@bford
Copy link
Contributor Author

bford commented Nov 9, 2022

Thanks for your comments. I haven't and probably won't have time to do an extensive analysis of large existing Go codebases to quantify how common this kind of need is. For the moment I'm just subjectively reporting that I've encountered this kind of need many times in a variety of contexts in recent years. Everyone's experience will vary, of course.

I agree that the "two types at once" perspective might be confusing and unnecessary. I'm not sure that I immediately understand your alternative "scoped to the writer field" formulation, though.

Actually, another way of thinking about this that might be conceptually simpler and cleaner is as follows. The fields of a struct still have only one type - the declared type - so the proposal changes nothing in that regard. Assignments to/from that field, and other operations using that field, still work as always. There are only two (afaik) small "point changes" in behavior:

  • When taking a pointer to the field and immediately converting that pointer to an interface reference (e.g., var w io.Writer = &m.writer, or using &m.writer as an argument to a function taking an io.Writer parameter), it is no longer just the methods in the field's underlying type that can satisfy the interface's demands, but additionally the field methods declared as part of the enclosing struct. Thus, a pointer to the field is immediately assignable to interfaces that a pointer to its underlying type might not have been assignable to. The Go compiler arranges whatever method dispatch magic may be needed to connect the interface's methods either to the underlying type's methods or the field's methods, as appropriate.
  • Normal method calls directly referring to the relevant field can invoke either the methods of the field's underlying type or methods attached to the field in this proposal. That is, other code in the package could invoke m.writer.Write() directly in the above example, not just via an io.Writer interface assigned from an &m.writer pointer.

If you take a pointer to a field with methods and assign it to a pointer variable referring to the field's underlying type, however (e.g., var w *struct{} = &m.writer using the first example above), then that pointer of course "forgets" that the object it points to ever had additional methods that the struct{} type doesn't, and the pointer variable isn't usable to call those field methods.

A more subtle question is whether that "lost information" should be dynamically recoverable or not dynamically. Maybe it's simpler to say no: if the information about the existence of the field methods is not "captured" in an immediate assignment to a suitable interface, it's lost in any resulting pointer and cannot be recovered by dynamic type casts or type switch statements. But in the "hidden helper type" implementation suggested above, the natural answer might be yes, since a dynamic cast or type switch might be able to detect that hidden type hiddenHelper supports a Write() method and recover access to it. I'm frankly not sure which of these potential semantic variations is preferable: the former seems simpler from a typing perspective, while the latter seems more powerful and not necessarily more difficult to implement. But this is a detail that should be considered in a design document if this proposal or something like it were to get to that stage.

@ianlancetaylor
Copy link
Member

Another approach to this issue might be #25860. Then EncodeThyself could return an interface with a method where the method is a closure that refers back to m of type *MainObject.

This proposal seems like a lot of machinery to accomplish something that we can already do, that thing being to provide an exported method on a type that is only accessible from within the type. It took us a while to understand what is happening, which suggests that the operation is rather complex for a language like Go.

Is there anything we are missing here?

-- for @golang/proposal-review

@DmitriyMV
Copy link
Contributor

Also #47487

@ianlancetaylor
Copy link
Member

Based on the discussion above, and the emoji voting, this is a likely decline. Leaving open for four weeks for final comments.

@ianlancetaylor
Copy link
Member

No further comments.

@ianlancetaylor ianlancetaylor closed this as not planned Won't fix, can't repro, duplicate, stale Feb 2, 2023
@golang golang locked and limited conversation to collaborators Feb 2, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge LanguageChange Suggested changes to the Go language Proposal Proposal-FinalCommentPeriod v2 An incompatible library change
Projects
None yet
Development

No branches or pull requests

6 participants