Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: Go 2: function overloading #21659

Closed
prasannavl opened this issue Aug 28, 2017 · 67 comments
Closed

proposal: Go 2: function overloading #21659

prasannavl opened this issue Aug 28, 2017 · 67 comments
Labels
FrozenDueToAge LanguageChange Proposal v2 A language change or incompatible library change
Milestone

Comments

@prasannavl
Copy link

prasannavl commented Aug 28, 2017

This has been talked about before, and often referred to the FAQ, which says this:

Method dispatch is simplified if it doesn't need to do type matching as well. 
Experience with other languages told us that having a variety of methods with
the same name but different signatures was occasionally useful but that it could
also be confusing and fragile in practice. Matching only by name and requiring
consistency in the types was a major simplifying decision in Go's type system.

Regarding operator overloading, it seems more a convenience than an absolute
requirement. Again, things are simpler without it.

Perhaps, this was the an okay decision at the point of the initial design. But I'd like to revisit this, as I question the relevance of that to the state of Go today, and I'm not sure just adding a section in the FAQ fully justifies a problem.

Why?

Complexity level is low to implement it in Go

Function overloading doesn't have to complicate the language much - class polymorphism, and implicit conversions do, and that with functional overloading does so even more. But Go doesn't have classes, or similar polymorphism, and isn't as complex as C++. I feel in the spirit of stripping to simplify, the need for it was overlooked. Because overloaded functions, are, for all practical purposes, just another function with a suffixed internal name (except you don't have to think about naming it). Compiler analysis isn't too complicated, and literally every serious language out there has it - so there's a wealth of knowledge on how to do it right.

Simpler naming and sensible APIs surfaces

Naming things is one of the most difficult things for any API. But without functional overloading, this makes this even harder, and in turn encourages APIs to be named in confusing ways and propagate questionable designs so much that its repeated use makes it perceived to be okay.

I'll address some cases, going from debatable to the more obvious ones:

  • Case 1: Go's standard library is filled with functions such as this

      func (re *Regexp) FindAll(b []byte, n int) [][]byte { ... }
      func (re *Regexp) FindAllIndex(b []byte, n int) [][]int { ... }
      func (re *Regexp) FindAllString(s string, n int) []string { ... }
      func (re *Regexp) FindAllStringIndex(s string, n int) [][]int { ... }

    I believe this is confusing and there is no way to know what the functions' intents really are, unless we read the docs. (This is a common pattern we've become accustomed to over-time, but still isn't convenient in any way). Even worse, by avoiding function overloading, the language itself is now unfortunately encouraging people to create APIs with insensible names.

    With function overloading, this could be

      func (re *Regexp) FindAll(b []byte, n int) [][]byte { ... }
      func (re *Regexp) FindAll(s string, n int) []string { ... }
      func (re *Regexp) FindAllIndex(b []byte, n int) [][]int { ... }
      func (re *Regexp) FindAllIndex(s string, n int) [][]int { ... }

    I think this is much nicer, and simplifies the API as its essentially now just FindAll and FindAllIndex. It can reduced, and this also has implications that go into auto-completion, and grouping it in the API docs, etc.

    The other part of the redundancy in this specific example relates to generics or variant types, but that's out of scope of this issue.

  • Case 2: The above example is actually one of the better ones.

    http package:

    func Handle(pattern string, handler Handler) { ... }
    func HandleFunc(pattern string, handler func(ResponseWriter, *Request)) { ... }

    We've used it long enough so that this API has been ingrained into us by habit as to what it does. But let's think about it from a pure semantic perspective for a moment. What does HandleFunc mean? - Does it mean "Handle the Func itself", or "Handle it using the func"?. While, the team has done what's best without the capability of overloading - It's still confusing, and it's a poor API design.

  • Case 3:

    Even worse:

      func NewReader(rd io.Reader) *Reader { ... }
      func NewReaderSize(rd io.Reader, size int) *Reader { ... }

    I'm saddened that this style even became popular and into the standard library. Does this really mean "create a new reader of size", or "create a new reader size" (that is, semantically - an integer)?. And this list doesn't just go on, it encourages user to utilize the same design which creates a nasty world rather quickly.

Better default patterns for APIs

Currently one of the patterns that's considered a good way to pass options into APIs is: "The functional options" pattern by Dave Cheney here: https://dave.cheney.net/2014/10/17/functional-options-for-friendly-apis.

While it's an interesting pattern, it's basically a work around. Because, it does a lot of unnecessary processing, like taking in variadic args, and looping through them, lot more function calls, and more importantly it makes reusing options very very difficult. I don't think using this pattern everywhere is a good idea, with the exception of a few that "really fits the bill" and keeps its configurations internal. (Whether it's internal or not, is a choice that cannot be generalized).

The most common pattern that I'd think as a general fit would be:

type Options struct { ... opts }
func DefaultOptions() { return &Options }
func NewServer(options *Options) { ... }

Because this way, you can neatly reuse the structures, that can be manipulated elsewhere, and just pass it like this:

opts = DefaultOptions();
NewServer(&opts);

But it still isn't as nice as Dave Cheney's example? Because, the default is taken care there implicitly.

If there was function overloading, it basically can be easily reduced to this

func NewServer() { opts := DefaultOptions(); NewServer(&opts) ... }
func NewServer(options *Options) { ... }

This allows me to reuse the structures, manipulate them, and provide nice defaults - and you can also use the same pattern as above for APIs that fit the bill. This is much much nicer, and facilitates a far better ecosystem of libraries that are better designed.

Versioning

It also helps with changes to the APIs. Take this for instance #21322 (comment). This is an issue about inconsistent platform handling for the OpenFile function. And I find that a language like Rust has a much nicer pattern, that solves this beautifully - with OpenFile taking just the path, and everything else solved using a builder pattern with OpenOptions. Let's say hypothetically, we decide to implement that in Go. The perm fileMode parameter is a useless parameter in Windows. So, to make it a better designed API, let's hypothetically remove that param since now OpenOptions builder handles all of it.

The problem? You can't just go and remove it, because it would break everyone. Even with major versions, the better approach is to first deprecate it. But if you deprecate it here - you don't really provide a way for programs to change it during the transition period, unless you bring in another function altogether that's named, say OpenFile2 - This is how it's likely to end up without overloaded functions. The best case scenario is you find a clever way to name - but you cannot reuse the same good original name again. This is just awful. While this particular scenario is hypothetical - it's only so because Go is still in v1. These are very common scenarios that will have to happen for the libs to evolve.

The right approach, if overloaded functions are available - Just deprecate the parameter, while the same function can be used with the right parameters at the same time, and in the next major version the deprecated function be removed.

I'd think it's naive to think that standard libraries will never have such breaking changes, or to simply use go fix to change them all in one shot. It's always better to give time to transition this way - which is impossible for any major changes in the std lib. Go still has only hit v1 - So it's implications aren't perhaps seen so strongly yet.

Performance optimizations

Consider fmt.Printf, and fmt.Println and all your favorite logging APIs. They can all have string specializations and many more like it, that opens up a whole new set of optimizations possibilities, avoiding slices as variadic args, and better code paths.


I understand the initial designers had stayed away from it - but I feel it's time that old design decisions that the language has outgrown or, is harmful to the ecosystem are reopened for discussions - the sooner the better.

Considering all of this, I think it's easy to say that a tiny complexity in the language is more than worth it, considering the problems it in turn solves. And it has to be sooner than later to prevent newer APIs from falling into the insensible naming traps.

Edits:

  • Changed a few words which have incorrectly set an offensive tone to the issue. Thanks to those who pointed out, and my apologizes to anyone, whom I may have inadvertently offended - was not my intention.
  • Added versioning point
  • Added optimizations point
  • Better formatting
@davecheney
Copy link
Contributor

davecheney commented Aug 28, 2017 via email

@prasannavl
Copy link
Author

prasannavl commented Aug 28, 2017

@davecheney - Solutions to make sure things are compatible, and the semantics of the language doesn't change beyond what it has to be, can be discussed. But I feel, it's only productive to do so, if the problem is acknowledged and there is a willingness for it to be solved. So far, I've unfortunately only seen deflection and being just marked as unfortunate rather than to even attempt to solve it.

One of the reasons for me to open this, is to indicate the critical need for the discussion on a reasonable approach to this.

That being said, I can think of a few ways - the obvious one being to pass the signature along - which might seem rather tedious, but the compiler can quite easily infer it - and in cases of ambiguity won't compile. These are "solvable" problems. But API design restrictions imposed by the language itself is not.

@cznic
Copy link
Contributor

cznic commented Aug 28, 2017

While the proposal author is completely entitled to any opinion, the form of its presentation makes me uncomfortable.

Random samples:

Function overloading doesn't really complicate the language...

Opinion, but presented like a fact.

I don't think anyone would call this as good API design in anyway.

Wrong, I do.

I think we can all agree that the naming is just horrific, ...

Wrong, I disagree.

We've used it long enough so that this ridiculous API has been ingrained in our brains as to what it does.

Using words like ridiculous with regard to someone's else work really doesn't help your case.

And so on.

@as
Copy link
Contributor

as commented Aug 28, 2017

Calling something badly designed is easier than demonstrating it by example. If I see FindIndexString, I know I have a string. In your proposal, I have to look further and continue using human cycles to determine which FindIndex I'm looking at. I generally read more Go than I write Go by an order of magnitude so that "String" suffix saves me quite a bit of time over the years.

I don't think the language will be improving as people start populating packages with overloaded functions. The proposal lacks a basis of its assertions that this design is ridiculous. The as-seen-on-TV tonality of the proposal doesn't help either.

@prasannavl
Copy link
Author

prasannavl commented Aug 28, 2017

@cznic My apologies. Disregarding some one else's work wasn't my intention. If I've come across so - I do apologize. However, my emphasis on using the words "ridiculous", and "horrific" weren't to disregard them - I had to emphasize on the limiting nature of the language and it's imposition on why such APIs have propagated to an extent that they're seen as okay. I've unfortunately come to see a lot dogmatic views on things in the Go community, without the willingness to see the problems for what they are in many cases.

I strongly believe that APIs such as this

    func NewReader(rd io.Reader) *Reader { ... }
    func NewReaderSize(rd io.Reader, size int) *Reader { ... }

are horrible APIs, when they're a part of the language standard library. They are highly misleading - I recognize that they were created due to such restriction in the language. These are not meant to disregard the work of those who created the APIs, but in order to insist on the restrictions of language that forced them to create so.

I do apologize if my words came out offensive, than to convey my intention correctly. Hopefully, we can look past that and into the actual problem. Cheers. :)

@prasannavl
Copy link
Author

@as, The semantic meaning of FindIndexString is find the index string, not find the index from a string (which is what the API means). This is my point, it has been so common for things to be misguided, that it has become OKAY to do so - which is not good.

@as
Copy link
Contributor

as commented Aug 28, 2017

You are missing the point. There is no FindIndexString. The function is regexp.FindStringIndex. A specific function is not relevant to the proposal.

@prasannavl
Copy link
Author

prasannavl commented Aug 28, 2017

@as, With reference to your readability remark - I would argue that reading a FindIndex conveys all the meaning, and even simplifies the code. It doesn't take anymore human cycles, as you already know that it find the index of the thing that you pass into it, regardless of what it is. The intent is very clear. And if you go ahead and change it to a byte representation above, the semantics are preserved as such with no code changes, which is an added advantage.

Though, even if you need to know the exact type - let's face it, we're not in the 80s. We have language tools that assist. Constraining a language because of these limitations seems counter-intuitive to me. Almost any decent editor with language support can take you right to the function. And lesser code that conveys intent is always cleaner. "FindIndex" is just lesser than "FindIndexString". And suffix 'String' semantically becomes an implementation detail, not just the intent.

@ianlancetaylor ianlancetaylor changed the title Function overloading proposal: Go 2: function overloading Aug 28, 2017
@gopherbot gopherbot added this to the Proposal milestone Aug 28, 2017
@ianlancetaylor ianlancetaylor added v2 A language change or incompatible library change LanguageChange labels Aug 28, 2017
@ianlancetaylor
Copy link
Contributor

I've relabeled this as a proposal, but in fact there is no full proposal here. Perhaps this discussion should be moved to the golang-nuts mailing list.

I understand that the goal is to first say that function overloading could be added to Go 2, and to then discuss how it might work. But Go language changes don't work that way. We aren't going to say it's OK in principle, and lets figure out the details. We want to see how it really works before deciding whether it is a good idea.

Any proposal for function overloading needs to address how to handle cases like

func Marshal(interface{}) []byte
func Marshal(io.Reader) []byte

in which for a given argument type there are multiple valid choices of the overloaded function. I understand that others will disagree, but I believe this is a serious problem with C++ function overloading in practice: it's hard for people to know which overloaded function will be selected by the compiler. The problem in Go is clearly not as bad as the problem in C++, as there are many fewer implicit type conversions in Go. But it's still a problem. Go is intended to be, among other things, a simple language. It should never be the case that people are confused about which function is being called.

And lest anyone produce a simple answer for the above, please also consider

func Marshal(interface{}, io.Reader) []byte
func Marshal(io.Reader, interface{}) []byte

@prasannavl
Copy link
Author

prasannavl commented Aug 28, 2017

@ianlancetaylor - Thanks. Correct me if I'm wrong, but I understood this to be more of a philosophical change than a technical one, and a controversial topic to start a proposal without some feedback.

Because, technically, as you said, Go doesn't suffer from the complexities of C++. In fact, both of the above that you stated can be easily solved with simply disallowing all implicit conversions while matching, and disallowing them to compile at all when in doubt of an ambiguity.

Very high-level matching:

  1. Match concrete types -> if they're exact matches - function identified.
  2. Match specific interfaces -> Check interface compatibility -> If compatible - refuse to compile, else function identified. (This disallows both the above cases, as io.Reader is compatible with interface{})
  3. Match to interface{}, if and only if the matches above didn't succeed or error, and such a match exists.

The above matches are always only entered when an overloaded method exists - so the compiler effectively only pays these costs when needed.

This should work nicely, because, a large number of scenarios covered above simply deal with concrete types.

Allowed:

func Marshal(string) []byte
func Marshal(byte) []byte
func Marshal(io.Reader) []byte

Allowed:

func Marshal(string) []byte
func Marshal(byte) []byte
func Marshal(interface {}) []byte

Side-note: All this opens up room for some very easy optimizations by means of specializations for APIs like "fmt.Printf" for strings, etc without unnecessary allocations.

Refuses to compile:

func Marshal(interface{}) []byte
func Marshal(io.Reader) []byte

@martisch
Copy link
Contributor

martisch commented Aug 28, 2017

Side-note: All this opens up room for some very easy optimizations by means of specializations for APIs like "fmt.Printf" for strings, etc without unnecessary allocations.

Would the suggested very easy optimization of Printf e.g. fmt.Printf(string, string) need to also duplicate and specialise the underlying interface{} functions that are used by fmt.Printf:
Fprintf, doPrintf, ... to not just allocate later?

@prasannavl
Copy link
Author

prasannavl commented Aug 28, 2017

@martisch - Not duplicate, but refactor. Of course, it will have to go all the way down.

Things beyond the type assertion for the string which happens today, will have to be refactored into its own tiny method, which the underlying implementation for the specialization will have to directly call.

@griesemer
Copy link
Contributor

@prasannavl There's no doubt that it's possible to come up with rules that somehow handle overloaded functions (C++ can do it...). That said, there really is a complexity argument as @ianlancetaylor has mentioned already. As it is, a function or method is uniquely identified by its name, it is a purely syntactic operation. The benefit of that simplicity, which you may consider perhaps too simple, is not be underestimated.

There's one form of overloading which I believe you haven't mentioned (or perhaps I missed it), and which also can be decided syntactically, and that is permitting the same function (or method) name as long as the number of arguments is different. That is:

func f(x int)
func f(x int, s string)
func f()

would all be permitted together because at any one time, depending on the number of arguments it is very clear which function is meant. It doesn't require type information (well, almost, see below). Of course, even in this simple case we have to take care of variadic functions. For instance:

func f(format string, args ...interface{})

would automatically consume all the functions f with 2 or more arguments.

Overloading based on argument count is like adding the number of arguments to the function name. As such, it's easy to determine which function or method is called. But even in this simple case of overloading we have to consider situations such as:

f(g())

What if g() returns more than one result? Still not hard, but it goes to show that even the simplest form of overloading can quickly lead to confusing code.

In summary, I believe for this proposal to be considered more seriously, there's a couple of things missing:

  1. A concrete detailed description of the overloading rules proposed with a discussion of special cases (if any).
  2. One or multiple concrete examples of existing code where overloading would solve a real problem (such as making code significantly easier to read, write, understand, or whatever the criteria).

@as
Copy link
Contributor

as commented Aug 28, 2017

Currently, the proposal seems to lack the engineering salience to seriously considered. Keep in mind that unless you are planning to create an experimental compiler and eventually working code, someone else from the Go team will have to implement this proposal.

In order to do that, there needs to be a perceived benefit that outweighs the cost of putting the change into the language. Ignoring the chaff (e.g., ridiculous, horrible), you submit that the language designers did not know what they were doing. Not really a good start for a proposal.

You need to prove your problem is everyone's problem. Then you need to provide a solution that stands up to technical scrutiny. Then the powers that be decide if that solution is simple and coherent enough to make it into the language.

@ianlancetaylor
Copy link
Contributor

I think you are suggesting that it will be impossible to overload functions based on different interface types when it is the case that one of the interface types is assignment compatible with the other. That seems like an unfortunate restriction. It's already always possible to use a type switch to select different behavior at run time. The only gain from using overloaded functions is to select different behavior at compile time. It is often natural to handle different interface types in a different type, and there are plenty of examples in existing Go code of doing that at run time. Being able to overload some types at compile time but not others seems to me to be a weakness.

I forgot to mention another problem you will need to consider when it comes to writing a proper proposal, which is that Go constants are untyped.

func F(int64)
func F(byte)
func G() { F(1) } // Which F is called?

@ianlancetaylor
Copy link
Contributor

Also it will presumably be possible to overload methods, not just functions, so it becomes necessary to understand the meaning of method expressions and method values when they refer to an overloaded method.

@cznic
Copy link
Contributor

cznic commented Aug 28, 2017

func F(int64)
func F(byte)
func G() { F(1) } // Which F is called?

Here the rule could possibly be to use the same default type constants have in short variable declarations, ie. F(int) (not listed in the example, so a compile error in this case)

@jimmyfrasche
Copy link
Member

@cznic consider this code

package foo
func F(int64)

package bar // in a different repo
import "foo"
func G() { foo.F(1) }

That works but then foo adds func F(byte). Now bar.G() no longer compiles until it's rewritten to use an explicit conversion.

There's also the case of embedding an interface an interface when they have overlapping overloads.

@cznic
Copy link
Contributor

cznic commented Aug 28, 2017

That works but then foo adds func F(byte). Now bar.G() no longer compiles until it's rewritten to use an explicit conversion.

Nice catch.

@mahdix
Copy link

mahdix commented Aug 28, 2017

In Go a function can be assigned to a variable or passed as a parameter. If functions can be overloaded, how will programmers indicate which variant of an overloaded function they meant?

@davecheney I think this can be resolved using the type of the variable, to which a function pointer is assigned. Can you give an example?

@davecheney
Copy link
Contributor

func Add(int, int)
func Add(float64, float64)

g := Add

What is the type of g?

@ghost
Copy link

ghost commented Aug 29, 2017

func (S) F1()
func f(interface{ F1() })
func f(interface{ F2() })

var s S
f(s)

This works until S implements F2.

@neild
Copy link
Contributor

neild commented Sep 6, 2017

I think that every time I've wished for some form of function overloading, it has been because I was adding a new parameter to a function.

func Add(x, y int) int {}

type AddOptions struct {
  SendMailOnOverflow bool
  MailingAddress string // default: 1600 Amphitheatre Parkway 94093
}

// AddWithOptions exists because I didn't think Add would need options.
// If I was starting anew, it would be called Add.
func AddWithOptions(x, y int, opts AddOptions) int {}

The cost of adding a ...WithOptions function isn't very high, so this generally isn't a huge concern for me. However overloading where the number of parameters must differ, as mentioned by @griesemer, would admittedly be useful in this case. Some form of default value syntax would also serve.

// Add(x, y) is equivalent to Add(x, y, AddOptions{}).
func Add(x, y int, opts AddOptions = AddOptions{}) in {}

@bcmills
Copy link
Contributor

bcmills commented Sep 7, 2017

To echo @neild's comment, it seems like all of these cases would be better addressed by either simpler or more general features proposed elsewhere.

Case 1 is really two issues: []byte vs string (which already has a cluster of related issues, exemplified by #5376), and Index vs. Value (which, as @cznic notes, is a reasonable API distinction).

Generics (#15292) would address the issue more clearly than overloading, because they would make it obvious that the only difference is in the types (not other behaviors):

func <T> (re *Regexp) FindAll(b T, n int) []T { ... }
func <T> (re *Regexp) FindAllIndex(b T, n int) [][]int { ... }

Case 2 is arguably better handled (ha!) by sum types (#19412):

func Handle(pattern string, handler Handler | func(ResponseWriter, *Request)) {
	...
}

But it's not obvious that even those are necessary: HandleFunc is already a bit redundant with the HandlerFunc type.

Case 3 is the default-parameter case that @neild mentions.

func NewReader(rd io.Reader, size=defaultBufSize) *Reader { ... }
r := bufio.NewReader(r, size=1<<20)

We can already fake it today with a combination of varargs and structs, and we would be even closer if we could easily use a literal type for the optional arguments (#21496).

func NewReader(rd io.Reader, opts ...struct{ Size int }) *Reader { ... }
r := bufio.NewReader(r, {Size: 1<<20})

@prasannavl
Copy link
Author

prasannavl commented Sep 9, 2017

@ianlancetaylor , @griesemer - Doesn't #21659 (comment) solve all of what you have mentioned? Infact, I think it pretty much addresses most of the following comments.

@griesemer - Just the number of arguments is a good compromise, except the simplicity is not going to be what meets the eye, due to what you had already mentioned in dealing with interfaces. However, a pattern like what I suggested is a set of rules, that makes it possible, while still retaining a good amount of simplicity.

The internal impl can look something on the lines of FunctionName_$(Type1)_$(Type2)_$(Type3)() { } when overloaded functions are found, at it's simplest compromising on the function names (though this can be numerically optimized later with a bit of added complexity).

@ianlancetaylor

I think you are suggesting that it will be impossible to overload functions based on different interface types when it is the case that one of the interface types is assignment compatible with the other. That seems like an unfortunate restriction.

Yes, it is restricted. But I see no reason that overloading has to be "all or nothing". I think concrete type specializations and interface{} with non-compatible interfaces would solve most uses cases without complicating things and the 'ick' factor associated with dealing with complex interfaces.

Besides, even if later found a necessity for, which I doubt, it should be far easier to go from a limited set, to an all encompassing set, but not the other way around. Learning from the other languages, searching for a perfect solution is probably not going to take us far - It's going to end up like the case of generics - 10 years and counting.

@bcmills
Copy link
Contributor

bcmills commented Sep 12, 2017

@prasannavl

#21659 (comment) reminds me that overloading is fairly closely tied to function naming, and especially to constructors. When you only get one name to work with for constructor functions — the name of the type itself — you end up needing to work out some way to squeeze lots of different initialization options into just that one name.

Thankfully, Go does not have distinguished “constructors”. The functions we write to return new Go values are just functions, and we can write as many different functions as we want to return a given type: they don't all have to have the same name, so we don't have to work as hard to squeeze everything into that name.

@bronze1man
Copy link
Contributor

function overloading is bad to read code for me in java and c#.It is difficult to find the implement code as you need to know the type of the function arguments.
I will not use function overloading in java and c#, but others will use it,so it just make write not readable code easier.

Here is my work around with the function overloading need(add arguments after the function has been called by outer caller):

type OpenRequest struct{
   FileName string // need pass this one
   IsReadOnly bool // default false
   IsAppend bool // default false
   Perm FileMode // default 0777
}
func Open(req OpenRequest) (*File,error){
   if req.FileName==""{
     return nil,errors.New(`req.FileName==""`)
   }
   if req.Perm==0{ 
     req.Perm = 0777
   }
   ...
}
Open(OpenRequest{
   FileName: "a.txt",
})

I just found that you can add almost unlimit number of arguments to this function, and the caller can pass some of arguments, and ignore other one, and the arguments is more readable from caller code.

@ghost
Copy link

ghost commented Sep 28, 2017

Here is my work around with the function overloading need(add arguments after the function has been called by outer caller):

Facepalm.

@SCKelemen
Copy link
Contributor

If we are considering function overloading, are we also going to consider pattern matching? In my opinion, they are related: The same concept implemented differently.

@ghost
Copy link

ghost commented Sep 28, 2017 via email

@quasilyte
Copy link
Contributor

quasilyte commented Oct 4, 2017

Have anyone mentioned that overloading will require additional* name mangling for Go symbols?
With generics on board, this is even more exciting.

(*) As far as I know, right now there is only package prefix prepended to the symbol.

@pciet
Copy link
Contributor

pciet commented Dec 16, 2017

context adds method repeats to the database/sql API: https://golang.org/pkg/database/sql/

func (db *DB) Exec(query string, args ...interface{}) (Result, error)
func (db *DB) ExecContext(ctx context.Context, query string, args ...interface{}) (Result, error)
func (db *DB) Ping() error
func (db *DB) PingContext(ctx context.Context) error
func (db *DB) Prepare(query string) (*Stmt, error)
func (db *DB) PrepareContext(ctx context.Context, query string) (*Stmt, error)
func (db *DB) Query(query string, args ...interface{}) (*Rows, error)
func (db *DB) QueryContext(ctx context.Context, query string, args ...interface{}) (*Rows, error)
func (db *DB) QueryRow(query string, args ...interface{}) *Row
func (db *DB) QueryRowContext(ctx context.Context, query string, args ...interface{}) *Row
...there's more for other types

@prasannavl
Copy link
Author

@pciet, great example! And we're only in v1 of the language. I'm scared of how these APIs will evolve without overloading.

@pciet
Copy link
Contributor

pciet commented Dec 16, 2017

For the context in database/sql case I’m not convinced function overloading is the right pattern to fix repeating APIs. My thought is change *DB to a struct of reference/pointer types and call methods on an sql.DB instead, where the context is an optional field assigned before making these calls to Exec, Ping, Prepare, Query, and QueryRow in which there’s an if DB.context != nil { block with the context handling behavior.

@neild
Copy link
Contributor

neild commented Dec 18, 2017

The database/sql case seems to me like an argument for better API versioning rather than for function overloading.

@prasannavl
Copy link
Author

prasannavl commented Dec 18, 2017

@neild, precisely. And overloading is one of the most helpful ways to achieve API versioning. (As mentioned in the initial post already)

The problem? You can't just go and remove it, because it would break everyone. Even with major versions, the better approach is to first deprecate it. But if you deprecate it here - you don't really provide a way for programs to change it during the transition period, unless you bring in another function altogether that's named, say OpenFile2 - This is how it's likely to end up without overloaded functions. The best case scenario is you find a clever way to name - but you cannot reuse the same good original name again. This is just awful. While this particular scenario is hypothetical - it's only so because Go is still in v1. These are very common scenarios that will have to happen for the libs to evolve.

This just so happens to be an example, similar to the hypothetical scenario I mentioned. It's not that you can't do it without, but you have to jump through hoops, possibly with new packages even, to correct mistakes of old.

@quasilyte
Copy link
Contributor

quasilyte commented Dec 19, 2017

@neild, precisely. And overloading is one of the most helpful ways to achieve API versioning. (As mentioned in the initial post already)

@prasannavl, as already pointed out, this is not fair solution as it can break existing code that assigns function referring to it by its name.
The whole "versioning" part can be misleading as overloadeding can be backwards-incompatible in unexpected ways (think about C code that calls your Go functions for example of less expected/popular example).

If you know the solutions to those complications or this is an acknowledged risk/tradeoff, please mention it in the first message.
Current solutions do not cause these troubles, so the alternative should consider that.

That being said, I can think of a few ways - the obvious one being to pass the signature along - which might seem rather tedious, but the compiler can quite easily infer it - and in cases of ambiguity won't compile. These are "solvable" problems. But API design restrictions imposed by the language itself is not.

I also don't get you point how function overloading helps tooling.

  1. It will get clumsier to "goto definition" of overloaded function because of candidates list.

  2. If there will be much smart inference from the compiler, public API for tools should be provided in order for them to use that information. Otherwise very few tools will adopt it properly.

  3. Tools that rely on the unique property of {package name}+{function name} combinations will break.
    And I would not presume that it is very trivial to fix all of those cases. It can be impossible to remedy by go fix.

I have a feeling that you underestimate the associated complexities at the whole picture.
Sorry if I am wrong, but things like "quite easily" or "Complexity level is low to implement it in Go"/"just another function with a suffixed internal name" are confusing.

offtopic

My understanding is that C++ resolutions are hard due to other reasons (templates, namespaces/dependent name lookup are better examples).

But Go doesn't have classes, or similar polymorphism, and isn't as complex as C++.

@prasannavl
Copy link
Author

prasannavl commented Dec 19, 2017

as already pointed out, this is not fair solution as it can break existing code that assigns function referring to it by its name.
The whole "versioning" part can be misleading as overloadeding can be backwards-incompatible in unexpected ways (think about C code that calls your Go functions for example of less expected/popular example).

Can be backwards-incompatible? Yes.
Should it backwards-incompatible? Not really.

The key here is in how it's implemented. Let's think about what the variables here are - It's only the function parameters. So, if we can come up with a way, where the generated function names are tethered to the function parameters, this can solved nicely. (I do vaguely remember mentioning something on these lines in one of the comments before). That said, I can imagine this being a big problem in a language like C where dynamic linking is extensive. In Go, a majority of the use cases are static and ABI compatibility can also be forgiving (though I don't imply that things should break).

Let take an oversimplified example (oversimplified because this can't work due to other problems yet to be solved discussed in the comments before)

func DoSomething(in string)
func DoSomething(in int32)

And it's internal implementation would look something on the lines of:

func DoSomething__In_String(in string)
func DoSomething__In_Int32(in int32)

This shouldn't break C calling Go, or Go calling Go, or binary compatibility.

I also don't get you point how function overloading helps tooling.

I think it'd be fair to just refer to C# here. The tooling around C# does exactly what I mention, and it's a language with one of the most stellar language services and tooling. (PS: It handles a lot more complexity that isn't needed in Go, as the language is far more complex).

It will get clumsier to "goto definition" of overloaded function because of candidates list.

Sorry, I really don't see how. Each reference directly is tied to one definition, or it's incorrect code that won't compile.

If there will be much smart inference from the compiler, public API for tools should be provided in order for them to use that information. Otherwise very few tools will adopt it properly.

Possibly. While I'd like to explore the possibility on how the "tax" from this can be reduced, I am certain the best approach in most scenarios is what you mention.

Tools that rely on the unique property of {package name}+{function name} combinations will break.
And I would not presume that it is very trivial to fix all of those cases. It can be impossible to remedy by go fix.

There are no cases that will break existing code. Go 1 tooling will remain compatible with Go 1 code. Hypothetically if Go 2 implements function overloading, the unchanged tooling just won't detect the new function that use overloading, which will of course require updated tooling.

@quasilyte
Copy link
Contributor

quasilyte commented Dec 19, 2017

Sorry, I really don't see how. Each reference directly is tied to one definition, or it's incorrect code that won't compile.

Well, maybe that point is not very important for others anyway.

In very-very short form: In Emacs (or any other editor/IDE, actually)
M-x find-function foo.Bar can't work without additional prompt if multiple foo.Bar exist. This basically boils down to the fact that you need two things instead of one to find the function: it's name and actual arguments (or their types).

@prasannavl
Copy link
Author

prasannavl commented Dec 20, 2017

@quasilyte - Ah. Thanks for that. I was thinking of only the goto-definition by pointing at a function use, and navigating to the definition from there. (F12 from the actual function usage, in VSCode for example)

Thinking of the scenario you mentioned, yes, an additional prompt would be needed, when appropriate. Though most tooling assisted editors I tend to use - VS Code, Gogland, etc have as-you-type fuzzy search anyway that boils it down where I wonder if this is even noticeable. (C# has it, and never felt it to hinder ease of use or the speed of navigation. So does C++, Rust (traits), JavaScript etc). But yes, I suppose it might involve an extra key-press, and some people value it a great deal than others - though I really really wish one wouldn't state that as an argument against overloading.

@quasilyte
Copy link
Contributor

quasilyte commented Dec 20, 2017

@prasannavl, did you answered #21659 (comment)?
Please, consider this case.
It's a technical detail, not subjective or "religious".

I am not a good advisor here, but maybe technical design document may be a better argument than repeating the ones that proven not to be working (not everyone will agree on "API just get's better").
You may browse existing design documents to get inspiration.

Can be useful: adhoc polymorphism in the context of Haskell
Not all kinds of polymorphism play well with each other.
If generics are desired, they should be somehow designed together.

@quasilyte
Copy link
Contributor

quasilyte commented Dec 20, 2017

There is a backwards-compatible way to introduce adhoc polymorphism into Go without some problems mentioned above though.

Introduce a new keyword that defines overloadable function.
This function can not be assigned assigned unless type elision is possible (see #12854).
No existing code is affected.

xfunc add1(x int) int { return x + 1 }
xfunc add1(x float32) float32 { return x + 1 }

f := add1 // Error: can't assign overloadable function
var f1 func(int)int = add1 // OK
var f2 func(float32)float32 = add1 // OK

func highOrder(f func(int) int) {}

highOrder(add1) // OK

xfunc is just a placeholder for a better (and new) keyword.

When overloadable function is assigned, it's type is concrete and it can't be distinguished from normal function.
The only differences are in the way functions are defined (to tag their names and store function info in separate data structure during compilation) and assigned.
No issues with "overloadable functions as parameters", because the value can't have a "overloadable function" type.

Overloadable functions share same namespace as normal functions,
hence it is not possible to have f as overloadable and ordinary function.

If considered in the context of original proposal goals, there is a downside.
Programmer must know beforehand which function may require overloading in future.

@creker
Copy link

creker commented Dec 20, 2017

@quasilyte I'm not @prasannavl but the answer is obvious - compile time error. If type inference doesn't work then compiler should generate an error. Go has var [name] [type] syntax to solve that. No need for new keywords or anything like that. Existing code will continue to compile as long as you don't start adding overloads. And when something breaks it will be trivial to fix by hand. But probably not something that can be fixed by tooling.

@quasilyte
Copy link
Contributor

quasilyte commented Dec 20, 2017

@creker, maybe it's my personal problem with "compatibility" term and "API versioning" solution.
If overloading is sold as a solution for that, why it makes it so easy to break code that is depending on the library?

This is why "compile error" is obvious, but suboptimal, in my opinion.
Also, some languages do it in other way and defer error until there is no way to infer the actual type.
These cases should be a part of proposal to avoid misunderstanding.
Obvious things are not exception.

@creker
Copy link

creker commented Dec 20, 2017

@quasilyte type inference can be as sophisticated as it can but when there's no other way then compiler should throw an error. The comment that you mentioned is ambiguous without proper context and should not compile. How exactly sophisticate it is should probably be in the proposal document. I agree with you on that.

About breaking client's code, that's definitely something to think about and should be covered extensively in the proposal. Even language like C# has problems with that. But it most cases examples of such breaking are more telling about bad library design rather than problem with overloading. MS adds overloads with every .Net release and doesn't break anything because if you design your API properly then adding an overload is not breaking change.

@bcmills
Copy link
Contributor

bcmills commented Dec 20, 2017

if you design your API properly then adding an overload is not breaking change.

That depends on how strictly you want to define “breaking change”. If someone, say, assigns a function to a variable, and later invokes that function by name, the two references may resolve to the same overload in one version and a different overloads in the next.

That pretty much implies that if you want a strong compatibility guarantee, you must prohibit users from ever referring to a specific overload, including by assigning it to a variable. The compatibility guidelines for Google's Abseil C++ libraries make that explicit: “You cannot take the address of APIs in Abseil (that would prevent us from adding overloads without breaking you).”

@creker
Copy link

creker commented Dec 20, 2017

@bcmills you could still design your API to solve that. If your overloads are ambiguous then it's your problem to make them compatible. Even if reference resolves to different overload client's code is still working.

Say, you had one function

func foo(interface{})

Now you add an overload

func foo(io.Reader)

Obviously client's code that passed io.Reader before will now reference the latter overload. It's your problem to retain compatibility between the two even if they're used interchangeably. If your overloads are so incompatible that you can't even solve that then it's obviously API design problem.

Another solution could be on the language level. Forbid referencing overloads without explicitly specifying the exact overload. In other words, f := foo will not compile in any case even if you can infer the type. Also, forbid type conversion between overloads even if arguments are compatible. In the case above, two overloads have different and incompatible types even though you can pass io.Reader as interface{}.

@prasannavl
Copy link
Author

prasannavl commented Dec 20, 2017

@prasannavl, did you answered #21659 (comment)?
Please, consider this case.
It's a technical detail, not subjective or "religious".

It just doesn't compile. If there's ambiguity you add signatures to compile. (But, this does suffer from the same compatibility issue. More on that below)

@quasilyte type inference can be as sophisticated as it can but when there's no other way then compiler should throw an error.

@creker, I think @quasilyte was referring to the part where the function that isn't overloaded, is then overloaded, that could prevent code that compiled before from not compiling? While this isn't really an issue at all when statically linking - it is a significant issue during dynamic linking.

Also since you mentioned C#, just want to point out here, while I think it's fair to compare tooling, I don't think it may be fair comparison to compare on a deeper level. Since C# has IL code in the middle, it provides with more flexibility that Go cannot achieve.

maybe technical design document may be a better argument

@quasilyte - I do agree. I think there's significant feedback that has been gained from this thread, to start thinking about a technical document. I hope to find time soon to collect things from this and potential impl possibilities into a document. 22/30 for/against a the time of this comment. Hmm.

PS: While I'm quite obviously advocating FOR overloading, if it HAS to introduce a new keyword, to me personally that would tip the scales, as that defeats the outer language simplicity enough for me to not purse it.

@bcmills - now, coming to the compatibility issue - thinking out loud here, there is one approach to make sure overloading doesn't break compatibility. But that'll break compatibility with existing Go. It's rather simple, but radical - change the internal repr of every Go function to have it's parameters as a part of the name

Eg:

func Hello(text string)
func Hello1(text string)
func Hello1(name string, text string)

Now the compatibility is an issue only when Hello, and Hello1 follow different methodologies for internal naming. But if you keep it consistent and change the internal representation of both to look like this

func Hello__$CMagic__string()
func Hello1__$CMagic__string()
func Hello1__$CMagic__string_!_string()

This solves compatibility. But introduces slightly more complicated debug tooling. (Not debugging, but debug tooling). The tooling will then have to actively convert the internal repr to human readable form of Hello1(name, text).

This does raise the cost of internal complexity more than I'd like. But does solve compatibility.

@ianlancetaylor
Copy link
Contributor

  1. Any approach here will add significant complexity to the spec. We will need to define something along the lines of the complex C++ rules to choose which overload is desired. It won't be as complex as C++ but it will be complex. One of the key reasons that Go code is easy to read is that it is easy to understand what every name refers to. This proposal will lose that property to some extent.
  2. In code like f := F where F is overloaded you will need to explicitly state the type of f, which is unfortunate.
  3. In code that uses method expressions that refer to an overloaded method, you will have to specify the type of the desired method, but how? This is a long issue but I don't see any good suggestion above for this problem.

Proposal declined.

@golang golang locked and limited conversation to collaborators Mar 20, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge LanguageChange Proposal v2 A language change or incompatible library change
Projects
None yet
Development

No branches or pull requests