Do you have a real world example that is improved by this feature?

@pciet

Well we have the package management problems with Go since day one.
Countless variants of "solutions" and the complexity involved.

If these dependencies could be avoided in the first place (and that's what would be inforced by this proposal) the better for us all.

The only unavoidable dependency is from the main package. If you write it, you "own" your imported external libraries, and you make sure they work. Only after a while, when updating, the problems arise.

Now if you have library package foo depending on type Bar of package bar and you also need package baz that is depending on type Bar of package bar that changed in an incompatible way, you have a problem. With this proposal foo and baz could not reference type Bar so there would be no dependency and no problem. Inside main you could easily wrap around bar.Bar to fullfill the incompatible interface of foo without any action from the developer of foo.

If neither bar nor foo may depend on the other, nor on a 3rd library, where is the definition of the common interface known by both foo and bar. Your proposal would only work with interfaces from the standard library.

@mbenkmann

If foo is meant to be used with bar, foo would define an exported interface type that is implemented by bar at the point when they are compatible.

BTW: This encourages a culture of similar interfaces for similar tasks within the community, so libraries can be swapped.

A good extension of that proposal would be to forbid init functions within libraries.
Then every initialization inside a library would be done via a function call from main.
That would make it more transparent, that there is initialization code and if this code changes there is a chance that the function name changes and such a change would get noticed at compilation time (compared to a change within init which would get unnoticed).

I appreciate the sentiment behind decoupling external libraries and the general idea of making the dependency graph wider rather than deeper, but this is unviable without covariant types, without generics, without disallowing packages that have side effects, and without much stronger type inference (and without surely many other things). I don't see how it would ever be possible in a language like Go.

This would make more sense in a purely functional language.

Since this is much too radical, I'd prefer some mechanism, tooling, or policy (or a combination of all three) that would encourage, or somehow help development of these "independent" libraries through some other means rather than by adding restrictions to the language.

@mbenkmann
The interface of foo does not have to be known to bar. This is the beauty of interfaces in Go.

You are assuming interfaces that do not include ANY custom types. How do you define useful interfaces for a graphics library if you don't even allow abstractions for Rectangle. You want those interfaces to use [2]int for a Point?

@4ad
Concerning generics it seems the Go team wants to have them in Go2.
Your statement is very broad. Can you give a concrete example where two packages could not be used in tandem with this proposal?

@mbenkmann
Why not? Or x,y int. For sure the libraries would look a bit different, but I question, that they would look worse. One would definitely make more use of builtins which would be another good side effect IMHO.

@metakeule I think @mbenkmann's last example is a good one. In general, to be really useful packages need to share data (through types), not just behavior (through interfaces). You can do everything solely with interfaces, but that doesn't make for a good programming model in a language like Go (Lisp would be fine here).

@4ad
I fail to see why that would result in worse programming. Decoupling is a good thing and it is why we have interfaces in the first place. And decoupling is far more important between packages. Where would we be, if not every package used io.Reader and io.Writer?

And what about a Dialog? With x,y,width,height, a title, a message text, buttons...?

I don't understand how this would work if library foo needs to build its implementation details on top of library bar. There is no main involved here. Someone is trying to provide others with a library that may be consumed by another library or a main. Your proposal would make it impossible to have private implementations that use existing libraries without asking some other person to pass you through an interface?

So let me make sure I understand this. If there is library "A" which wants to use some sort of embedded kV database or cache, it would instead code against an interface and ask the consumer of this library to pass it in? Now let's say you have someone else with library "B" that uses "A" and third person with main that uses "B"

main -> B -> A

According to your proposal, main would have to import the transitive key value db dependency, and pass it through to B as an interface, which would then have to pass it though to A as an interface?

I'm very lost as to why we would want this situation.

Maybe we should just exchange []unsafe.Pointer in our interfaces. Yay. Who needs type safety.

@mbenkmann
It has nothing to do with type safety.

package foo

type Rect interface {
   X() int
   Y() int
   Width() int
   Height() int
}

func UseRectangle(r Rect) {
...
}

package bar

type rect [4]int

func (r rect) X() int {
return r[0]
}

func (r rect) Y() int {
return r[1]
}

func (r rect) Width() int {
return r[2]
}

func (r rect) Height() int {
return r[3]
}

func MakeRect(x,y, width, height int) rect {
   return rect{x,y,width,height}
}

If the Rect interface would be used and offered by other libs you could combine them easily
(not possible with depencies on structs).

Perfectly type safe. (an [4]int would also be type safe BTW)

@justinfx

Yes you did understand correctly.

We would want it for 3 reasons:

• It makes the dependencies visible to the user. There are there anyways. It is like error handling: You can hide it behind exceptions or you pass the error and have it visible while having to deal with it. The end user would have to deal with the dependencies anyway when there are issues with compatibility and that leads to avoidance of too many depencencies
  (the proposal would encourage library developers to not depend on too much functionality of external libraries, since that would make their lib unattractive).
• The most important part is, that the libraries can be developed indepently from each other (which happens anyway whether we like it or not), but the user has a chance to fix compatiblity issues independently from the developers of the libraries he uses.
• No dependency hell.

So you are suggesting that instead of depending on a 3rd library implementing standard data structures, every library should contain a copy of the relevant code. Look at how much code your package "bar" needs just for a silly little class like rectangle. Take a real world example:

https://godoc.org/github.com/veandco/go-sdl2/sdl

Now I want to offer a sprite library. You're telling me I can't use sdl.Rect. I have to copy the code into my library, standard code for computing intersections, unions etc. of rectangles. Your suggestion comes down to NOT USING LIBRARIES.

@justinfx

func main() {
   b.New(a.New(kv.New()))
}

@metakeule yuk. So that means the main has to now be aware of how to initialize library B with the transitive library A dependency. And you have to do this for every dependency that library A wants to use, which means every library in between has to expose an injection point.

Furthermore, it means that if library A wants to use a 3rd party dependency, it has to now create its own interface definition to match that dependency. But there is nothing to say that any other similar suitable replacement will conform to that interface you have just invented for exposure.

@mbenkmann

The sdl library would have to be rewritten in order to be useful for other libraries with this proposal.
(e.g. offer methods on Rect etc.)
But since we are talking about Go2, it would probably have to be rewritten anyway.
And it would make perfectly sense to make a sprite library independant from a specific sdl implementation, doesn't it?

Okay, so we've reached the point where every existing library has to be rewritten to be used with Go2.
Not gonna happen. Go2 will be compatible with existing code.

@justinfx

What do you mean with "conform to that interface"? Do you mean "conform to the semantic of the interface", because otherwise the compiler tells you...

@mbenkmann
Not every, but a lot. But nothing from the standard library.

Also I heard, Go2 should be able to import Go1 packages. So there would be a way to distinguish them and the rewrite could be done incremental.

@justinfx
Also that is just like normal Go interface work. Never had an issue with it. I guess the trick would be to avoid relying on 3rd party libs whereever possible (which is also a good practice today) and make the exposed "API" as small as possible. Probably be making use of builtin types where possible. Mind you: that is just in between packages; inside a package hidden structures could have all the custom types.

Whatever, I think it is an interesting thought experiment. We can see, if something useful arises from it.

What I meant is that if I use an external library Foo in my own library, I have to spec an interface for it so that the chain of dependers above me can supply me with an implementation. Now let's say there is exactly one existing solution for Foo. The interface I spec out basically says "I know there is nothing out there besides Foo to match this. Please just pass me Foo so I can work. And let's hope that something other than a mock will also conform to this interface that I have now been forced to expose as my public api.

My point is that your suggestion turns a private implementation consuming a private dependency into the requirement to expose a public interface so that dependers can pass you everything you need. Your example of passing A to B in main just illustrates how transient types now have to be leaked into the main. Before, main never needed to worry about the private types of B. Now your main has to reach into all the dependencies to pass through chains of types. Yes I would have originnaly seen all the dependencies listed in my dep manager lock file, but I never had to concern myself with their apis. Now I would have to chain them up to satisfy that embedded cache implementation that I didn't know I had to think about, which exists two levels of dependencies away from my main.

The goal is noble, to try and force people to limit their use of external dependencies in libraries, but it seems this solution is meant to make it annoying and gross to even use external deps in a library so as to deter people from doing it.

@justinfx
Ok, before we can agree to disagree, let me just point out that IMHO what you call a "private dependency" isn't a real private dependency since its crossing the package borders.
That becomes apparent if the dependency breaks and 3rd party users are affected. Then it is not private anymore and the user of your code needs to dig through your code in order to understand what the problem is.
A real private dependency can IMHO only be within a repo and packages and subpackages (which could depend on each other without restrictions according to this proposal).

With bubbling up the loose behavioral dependencies to the main package, you pay the price of importing packages in the beginning (and it could be a deciding factor which library to use). I think this is more adequate then paying the price / biting in your ass after months or years when you are in maintenance mode and on other projects. Also keep in mind that the more dependencies you have (= your project gets larger), the more likely any version conflicts are, some of them might not even be solvable.

I prefer to know my risks upfront.

@justinfx It would be the duty of the package expecting a certain semantic from an interface to document the expectations. Also to offer example code for integration, so that the user can simply copy the code and does have to figure out the dependency by searching/looking up.

So in your example, package a would have the example glue code:

package main

import 'kv'
import 'a'

func main() {
   a.New(kv.New())
}

and package b would have the example glue code

package main

import 'kv'
import 'a'
import 'b'

func main() {
 b.New(a.New(kv.New()))
}

so users of package b would not need to look up the example glue code from package a but could just copy the example glue code from package b straight away.

It is just a question of culture and documentation.

The problems described here: https://sdboyer.io/vgo/failure-modes/
(diamond problem) could be completely avoided.

This is impossible with Go as it stands, for one main reason - interface{Get() Interface} is not satisfied by Get() Implementation. This means main has to write, each time, wrapper code between almost all packages. Here's a simple example using hypothetical "MySQL" db package and "Sqlx" package.

https://play.golang.org/p/zYaBPs6nuKz

package main

import "fmt"

// MyDB.GetTransaction has no way of returning a SqlxTransaction.
// Write wrapping code to allow them to work together.

type DBWrapper MyDB

func (d DBWrapper) GetTransaction() SqlxTransaction {
	t := MyDB(d).GetTransaction()
	return &t
}

func main() {
	fmt.Println( Get(DBWrapper(MyDB{}), "nineteen characters") )
}

// package sqlx

type SqlxDB interface {
	GetTransaction() SqlxTransaction
}

type SqlxTransaction interface {
	Get(string)
	Commit() int
}

func Get(db SqlxDB, key string) int {
	t := db.GetTransaction()
	t.Get(key)
	return t.Commit()
}

// package mysql

type MyDB struct {}

func (m MyDB) GetTransaction() (tran MyTran) {
	return 
}

type MyTran string

func (m *MyTran) Get(key string) {
	*m = MyTran(key)
}

func (m *MyTran) Commit() int {
	return len(*m)
}

The only realistic solutions to this are: change Go to allow methods to satisfy an interface if their return values satisfy the interface; always return interface{} and do typecasting in sqlx; or have sqlx provide "wrapper.go" that must be copied into the main package to allow it to actually work correctly.

Why should everyone be forced to code in such a manner and prevent almost all Go1 packages to be upgradeable to Go2 without manually rewriting? This will essentially split the ecosystem and force people to choose between using external packages and having the benefits of Go2.

I do not see a reason why this cannot just be a opt-in flag or a third party tool to warn against importing external packages.

@AlexRouSg
I think it would just have the benefits, if it is enforced. Maybe a translator could be written that would automatically rewrite library code, so that just the glue code must be rewritten.

@metakeule

If it is possible to create such a tool, then why can you not simply fork the packages and use the tool on it thereby only enforcing it on your packages/programs?

Or why can't people just release 2 packages, one normal and one translated?

Go 2 must be largely if not perfectly compatible with Go 1. A change that breaks pretty much every existing Go package and a good chuck of existing Go documentation is basically a non-starter for Go 2.

If I'm reading the proposal correctly, the main package is in charge of importing essentially every top level package that it uses, and is, further, responsible for somehow hooking them up. If package "a" needs values created by package "b", then the only wait it can get them is if the main package calls "b" functions to create them and passes them to "a". Requiring the main package to do this in all cases seems impossible awkward.

I want to point of that this proposal implies some implicit ordering between example.net/foo/bar/baz, example.net/foo/bar and example.net/foo. IIUC you propose that example.net/foo can import example.net/foo/bar/baz as a non-external package, but the other way around example.net/foo/bar/baz can import example.net/foo only as an external package. In other words, you are introducing restrictions on who and how can a package import based on the syntax of the import path. (internal also restricts who can import what using import path syntax, but it simply disallows some imports, it doesn't change the normal rules about the nature of a package, the how.)

While perhaps some sort of distinction between external and non-external packages is warranted (I am not convinced) making this distinction based on the fact that import paths appear to be hierarchical is a deep departure from the way Go works today (internal and vendor notwithstanding). Today, package topology is undetermined by the apparent hierarchy present in import paths (in fact this independence is a sometimes a pain point for newcomers to the language, who expect it). foo/bar can import foo, but just as well foo can import foo/bar (but not at the same time), and people make frequent use of this. Your proposal forces people to chose only one possible option.

In fact, foo/bar and foo might not even be related at all. In general, for clarity and simplicity people try (more or less) to keep the import path hierarchy related to the actual dependency graph, but this is not required. foo/bar might be more related to quux than to bar and quux/baz might be more related to foo than to quux.

To summarize, to a good enough approximation import path syntax merely tells us how to find a package. This includes special cases like internal and vendor which add special rules but still are about how to find packages (or whether to find them at all). You propose that the import path syntax would have additional meaning than simply telling us how to find a package, and this meaning you are proposing is incompatible with the way Go is used today.

Besides the compatibility problems, this proposal would make the language almost unusable for all but the simplest of programs; programming at scale usually requires composing libraries, which this idea intentionally makes difficult and annoying.

@4ad Well that was for practical reasons, but probably the rules could be harmonized with the rules existing today for internal packages (which have an internal path element).

However it is important to be able to distinguish de facto at least on the repo level.
Since go import paths are just paths, that may happen to be URLs there is nothing within the import path indicating the top level of a repo (or the "project boundary"). But the compiler should not need to check repos and also a repo should not be a requirement.

Because of this and of the nature, how code hosting plattforms like github are typically organized, the hierarchie seemed reasonable. But that is not the core of the proposal, if there is a better way to find out project boundaries.

@dpinela
What you are saying is the first what comes to mind, if one is exposed to such an idea.
It is a bit annoying. But it depends how the code is organized. An the real challenge is not to dismis such a proposal, but to try out, if and how code could be written at large with such restrictions in a way that minimizes the annoyance and to have a look, how much annoyance we are left with. I written some libraries this way and used them and it was not a big deal and I am not convinced that it wouldn't be worth the effort.

Everybody interested in this question "Could we avoid 2nd level dependencies completely with Go" make your tests, rewrite some code the way I suggested or come up with own ways and share your results.

@AlexRouSg
Sorry, I've the impression that you missed the entire point of the proposal.
I don't even know where to start...

@ianlancetaylor

I really appreciate the effort of the Go team to keep the compatibility promise of Go1.
However I think, it is a big mistake to determine that Go2 will be compatible to Go1 at this point where it is completely unclear what Go2 will be.

There is the great chance to revise some unfortunate design decisions of Go1 that would be totally missed with this commitment.

I am not expecting that this proposal is followed, however it shows a way to get rid of 95% of the package management problems we are facing and that let to complex tools like dep and similar tools (I am talking about complexity in code and workflow in difficult situations).

I think, this proposal is worth recognizing and maybe you get some inspiration that may influence some decision for Go2 that would result in lesser dependency problems.

I wouldn't throw it out of the window just because you can and there are so many objection from the standpoint of an existing eco system.

The proposal is an thought experiment (like every proposal?) and shows, that it is theoretically possible with the current Go1 to avoid 2nd level dependencies completely (apart from the standard library). Admittedly in a provoking form - as a proposal for Go2 - but with the intent to influence/inspire the design of Go2 in some non predictable way.

@dpinela

programming at scale usually requires composing libraries, which this idea intentionally
makes difficult and annoying.

In a surprisingly large number of cases, you can replace method definitions with top level functions that receive the needed object as normal argument. Now if your then top level functions have only a single purpose, they mostly need not that much behavior
from an object. That behavior could easily abstracted via an minimal interface.

If this approach is taking by the immediate libraries and the low-level ones, it should not be
hard or require much boilerplate to compose libraries.

This is not a proposal that I like. I can appreciate that it is good to avoid 2nd level dependencies. There are times that using 2nd level dependencies saves time and makes development much easier, especially when creating reusable code shared by multiple teams.

If you choose to use this proposed standard for your code, I have no objection as you can restrict what you do without restricting what I do. If you are proposing that I have to do the same, then I object, I am a very lazy programmer. In fact my laziness is why I became a programmer. I will spend all day automating a one hour job.

If I am publishing, then I am likely to vendor any 3rd party packages without semver. Or any 3rd party package that I feel needs a quality review. Packages published by my team members, I do not vendor because I am already a stakeholder supported by the programmer.

I do thank you for offering some very sound advice. Developing useful abstractions is always a bit of a challenge.

@comaldave
I respect your opinion. Just saying:

Go did already teach us in lots of places (e.g. code formatting, error handling, missing of generics, static compilation) that restrictions can lead to freedom in the end. The idea is followed here.

Ok, I've identified a real bummer:

With this proposal, it is not possible to create libraries that are easier abstractions over functions in a low level library, even if direct dependency on the low level types could be avoided. That means, use friendly simplifying libs have to be part of a low level project/repo and vice versa - which would be good from a maintenance and user viewpoint but is typically a social problems since the folks preferring low level work are typically not good at creating simple easy to use APIs and vice versa.

It is an interesting challenge to find a solution for this social problem.

Also, it prevents you as a user to organize your glue code and own project internal abstractions over 3rd party libs in project specific libs.

So maybe we should allow packages that have no domain name as part of their path to import anything and only restrict the ones that have domain name (and are therefor considered to be "published"). Then the standard library exception would automatically included in that definition. Also mono-repos should get no restrictions as long as they choose their pathnames properly.

See my UPDATE section in the proposal.
This way an abstraction could be build internally and when it is finished, a pull request could be made to include it in the low level repo for general consumption (to solve the social issue mentioned above).

I don't see why this proposal would address package dependency issues. Just because all interactions between packages must be mediated through the main package does not mean that a package can not change to being incompatible.

Seems way too Object Oriented... Also, nobody wants to write 500 interfaces before they start their code.

For instance, a Slack bot. In order to use a 3rd party slack library, I'd need to write an interface for Users, Groups, Channels, Messages, Emojis, etc. It defeats the purpose of using a library.

I don't want it to be in the main package though, since if I wanted to make a Twitter AND a Slack bot (managed by a single app), I don't want my twitter bot and slack bot to be in the same package. And as I said earlier, I don't want to write tons of interfaces before I get to write my program. It's just boilerplate.

@ianlancetaylor

1. You get no diamond dependency problem.
2. Problems could always be fixed by the user in main.
3. No need to fork and change the upstream library.
4. No need to dig deeply into an upstream library in order to be able to fix it, since the dependent parts are visible through the interfaces.
5. In combination with the removal of init functions:
   • Freedom to include different version of a library side by side.
   • Freedom to make wrappers between incompatible types of two versions of a library.
   • Freedom to fine tune initialization order.
     ....

Even if we assume that the basic idea of this proposal is a good one, it's still extremely impractical.

If I develop library lib which calls pkg.Helper, then I would have to test this against two, three, or more different Helper implementations. If someone would report a bug then it can be very hard to track down, as it may depend on the specific implementation of the Helper function they're using (it could have a bug, different assumptions, etc.) Even worse, it may depend on the interaction between pkg.Helper and otherpkg.Helper.

Dependency injection with interfaces can be very useful, but like most good ideas it becomes a bad idea when pushed to the extreme and applied to every single case. This is no exception.

Really interesting proposal, but it feels like handling of the issue is happening in the wrong place. Developers already have the ability to avoid importing packages outside of the standard library, simply by not importing packages outside the standard library. Which means, if that is already happening, then developers have a reason (good, bad, or horrible) for doing so.

In the packages I develop, the code imports various flavors of other packages:

• Utility libraries - "assert" and "errors" packages come to mind - build upon what's available in the standard library, and make certain use-cases easier.
• shared models and interfaces - packages that define structures used elsewhere
• large implementations of other people's APIs for which maintaining interfaces would be impractical - for example - Go implementation of the GitHub API.
• implementation details matter - using a specific implementation of an API to extract information not available by default - in the context of database drivers, extracting driver-specific error information for recovery logic or testing.

There are good reasons for all of these types of imports. Since the act of adding an export is an explicit act of coding, developers already have the opportunity to choose not to do that.

Treating the "stdlib" as somehow blessed assumes that the majority of Go code is being written for open-source consumption. In practice, who knows how much is written for private use, and in the context of those private uses, companies may build up their own extensions to the standard libraries. The Go team's appropriate reticence to add to the standard library makes this scenario very likely.

On top of that, the "main" package isn't really that privileged. When building larger projects, I've found a fairly logical approach is to build a "main" package that doesn't have much logic in it, and mostly accomplishes its work by delegation to a different package in the project. If nothing else, the existence of the library package that encompasses the functionality of the "main" package means that it is possible for downstream users of the "main" program to instead invent their own version of "main" based on importing that other library. However, for ease of implementation, that library package called by main is going to include concrete packages that it depends upon, not build up a whole additional layer of interfaces. That would be extra work that any sensible developer would try to avoid.

Experience shows that smaller interfaces are better. However, the approach of forcing everything into interfaces necessarily would lead to an increase in the average number of functions defined in an interface. These would not be well-designed interfaces, as they would, in many cases, simply be substitutes for large lists of functions defined on structures. I'm fairly certain someone clever would build a tool that would automatically define an interface in package A that matches all the function signatures of a structure in package B. In short, forcing this approach in libraries would likely lead to badly designed interfaces that still happen to be tightly coupled to implementations.

Overall, I'm intrigued by the design aim of the proposal, but the solution seems to miss the mark. I think it doesn't actually fully solve the intended problem, introduces a whole bunch of new ones, and would end up with too many poor interfaces. This calls for alternate solutions - perhaps options such as these:

• Improvement to linting tools to identify places where narrow interfaces could be substituted for concrete types (I'm a huge fan of gometalinter)
• Improve lint to flag packages that lead to diamond dependency problems. That is, library package A could import library package B, so long as library package B doesn't itself import packages outside of the standard library. Allowing the one level of external package reference would allow for all the most common use-cases, and generate pressure to introduce interfaces when the complexity increases.
• Adding "flavoring" to import statements - perhaps a "no-structs" import tag - which would hide all concrete structures from being imported, and only allow for constants and interfaces. Or perhaps a "function only" tag (so packages like "assert" could work.)

In general, to align with the totally practical approach that Go has taken, of requiring a demonstration of the the value of the proposal. In this case, perhaps by way of using some sort of linting to catch and discourage the specific presumed bad practice. When experience has borne out the value of eliminating the presumed bad practice, then go ahead and consider it for introduction in the language.

Experience shows that smaller interfaces are better. However, the approach of forcing everything into interfaces necessarily would lead to an increase in the average number of functions defined in an interface. These would not be well-designed interfaces, as they would, in many cases, simply be substitutes for large lists of functions defined on structures. I'm fairly certain someone clever would build a tool that would automatically define an interface in package A that matches all the function signatures of a structure in package B. In short, forcing this approach in libraries would likely lead to badly designed interfaces that still happen to be tightly coupled to implementations.

Yeah, that might well happen.

In general, to align with the totally practical approach that Go has taken, of requiring a demonstration of the the value of the proposal. In this case, perhaps by way of using some sort of linting to catch and discourage the specific presumed bad practice. When experience has borne out the value of eliminating the presumed bad practice, then go ahead and consider it for introduction in the language.

I encourage everyone interested to try this out and see how far you can get. I got pretty far in my experiments, but obviously that would require a bunch of new lib designed with a different mindset.

We aren't going to do this. It might possibly have been practical several years ago. Today it would just break all existing Go code, with no simple path forward. It's infeasible.