Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: make major versions optional in import paths #44550

Closed
pkieltyka opened this issue Feb 23, 2021 · 258 comments
Closed

proposal: cmd/go: make major versions optional in import paths #44550

pkieltyka opened this issue Feb 23, 2021 · 258 comments

Comments

@pkieltyka
Copy link

pkieltyka commented Feb 23, 2021

Semantic Import Versioning (SIV) is a novel idea for supporting multiple versions of a package within the same program. To my knowledge and experience, it's the first example of a strategy for supporting multiple versions of a project / dependency in the same application. More importantly though, its clever introductory design allowed it to offer multi-versioned packages in a Go program while maintaining Go's compatibility guarantee.

Multi-versioned packages in a single program can be quite powerful -- for instance, imagine a Web service where you'd like to maintain backwards compatible support for your consumers, you can simply use an older import path from the same origin repository and versioning control, and quite elegantly continue to support those older API versions.

Although SIV may be an elegant solution in the scenario described above, it also adds unnecessary complexity, cost, code noise, discoverability and ergonomics for the majority of packages (publicly and privately) which may not ever have a mutli-version requirement (I'd argue most packages, and simply we can look to other ecosystems to see this is true). I am sure the Go team has heard a lot of feedback on the friction of SIV. https://twitter.com/peterbourgon/status/1236657048714182657?s=21 and https://peter.bourgon.org/blog/2020/09/14/siv-is-unsound.html offers some excellent points as well.

Clearly there is a case for SIV as an elegant solution for supporting multiple versions of a package in a single application, and there is also a strong case to make SIV optional.

It's clear to me there is a design trade-off at hand, and there is no single correct answer. As I consider the 80/20 rule in making an architectural decision between two trade-offs of capability and usability, I prefer to go with the 80% case so long as it doesn't forego the 20% ability. Which design is the best to optimize for if we can still support both? In the case with Go today, its not possible to opt-out of SIV, or opt-into SIV -- I believe both approaches can yield a happy solution. If we were starting from the beginning, I'd suggest to have SIV be opt-in, but maybe at this point its better for it to be an opt-out design to maintain backwards compatibility with history.


I'd like to propose a path to make SIV opt-out at the level of an application developer consuming a package, while being backwards compatible with current packages and tools.

I'd like to use https://github.com/go-chi/chi as an example for this proposal which adopted semver ahead of Go modules and SIV, and is built for developer simplicity and ergonomics intended for pro Go developers, but also making it familiar and accessible for developers who are new to Go -- these are my design goals for chi as an author and maintainer as started back in 2017. My present goal is to release Chi v5 without a SIV requirement and the only way I can do so is with the proposal below:


Proposal, by example:

github.com/go-chi/chi/go.mod:

module github.com/go-chi/chi/v5

go 1.16

then, git tag chi as v5.0.0 to make the release.

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

In the above case, we're specifying the go.mod as expected with current behaviour with SIV from a library perspective. However, from the application perspective when fetching or consuming the library, I may opt-out of the "/v5" suffix in the import path and only adopt it in the scenario when I'd like to support "/v5" and "/v4" (or some other prior version), where I require the handling of multiple versions simultaneously in my program.

I believe the implementation of the above to be backwards compatible as developers would continue to use "github.com/go-chi/chi/v5" with older version of Go as SIV is implied, but optionally developers could make their choice of multiple-version support for the package by handling the import paths themselves and import "github.com/go-chi/chi" to utilize v5.x.x as specified by go.mod.

I believe changes to the Go toolchain for such support would be minimal and would be isolated to the components which build module lists.

Thank you for reading my proposal, and its consideration.

@peterbourgon
Copy link

Application developers may consume the package via go get github.com/go-chi/chi@latest

I believe this would represent a breaking change, as it is currently (?) parsed as the latest minor.patch version of major version v0 or v1. That's an artifact of what is I think the major problem with any form of optional SIV: the unversioned module path is, unfortunately, interpreted not as "no version specified" but instead as major version 0 or 1.

@theckman
Copy link
Contributor

theckman commented Feb 23, 2021

@peterbourgon I'm not sure the Go toolchain has ever been covered under the compatibility guarantee, based on other "breaking" changes that have been made. The guarantee explicitly says

Compatibility is at the source level.

@theckman
Copy link
Contributor

theckman commented Feb 23, 2021

I'm currently a maintainer of the https://github.com/PagerDuty/go-pagerduty package, which was also incepted before Modules and never had 0.x releases. This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

As of now I'm planning a v1.5.0 breaking release to work around SIV not being optional. There is a bug that makes the current API not work reliably (two conflicting fields trying to be unmarshaled into), and removing the erroneous field may result in some people needing to update their code to use the .ID field instead of .Id.

I cannot in good faith justify forcing people to update all of their files that use our Go Module with the SIV for a one character change that they may use in one file. I'm sorta justifying the breaking change to myself because the API has never been stable due to the conflicting field names.

It would be ideal if the SIV component of the import was optional, if only one major version of a dependency is present in the go.mod file. That way folks could update their dependency, and only update the actual files that need to be changed. Likewise, we are still able to support more complex projects who need to roman-ride the major versions while doing the transition.

I genuinely believe it would be a great idea for us to really consider this proposal, and that it would be a nice improvement to the user experience of the Go Toolchain and Modules. I am confident that the need to roman-ride between two major versions of a package will only be needed by a small subset of the larger Go Ecosystem, and so I wonder if it makes sense to make it a requirement for all. I believe accepting this proposal would decrease the cognitive hurdles that may need to be overcome when trying maintain a Module, while also giving us the ability to be flexible for those who really need it.

Edit: Instead of reacting with 👎 this comment, could you comment below quoting what you disagree with and why? I think that would result in a much more constructive interaction.

@theckman
Copy link
Contributor

@nbys I see you 👎 on my comment. I'm sorta gonna put you on the spot and ask if you can explain what part of that you disagree with and why you disagree with it?

@peterbourgon
Copy link

I am confident that the need to roman-ride between two major versions of a package will only be needed by a small subset of the larger Go ecosystem . . .

And even for that tiny fraction of the ecosystem, the requirement typically exists only transiently, during a dependency upgrade process that requires multiple steps. And this might be worth stating explicitly, because I'm not sure it's understood by all the stakeholders: codebases which need to allow multiple major versions of a dependency in a single compilation unit in order to make a complicated upgrade tractable are pathological, not normal. Getting into that state isn't an inevitable outcome of writing software, it's a product of a specific set of conditions representing a super-minority of projects in the overall ecosystem.

@thrawn01
Copy link

I'm the maintainer of https://github.com/mailgun/mailgun-go and I do understand this pain. The process of updating the mailgun-go library to v4 involved not only updating all the import paths but also every single golang code snippet in the mailgun documentation. This was a huge under taking that we don't wish to do again.

Our hope is that #32014 will get approved and standard tooling will exist to make upgrading import paths simple for both library devs and users. While this doesn't solve the need to update all our example snippets it should help ease the some of the pain that library maintainers have in this manner.

@pkieltyka
Copy link
Author

Respectfully, #32014 is not a real solution to solving the challenges with SIV. As you said yourself, all of your documentation had to change as well, and that is just the beginning of the permanent tax on developer experience that SIV imposes for a niche use-case of multi-version support in a single program.

@bcmills
Copy link
Contributor

bcmills commented Feb 23, 2021

@theckman, I commented on your specific example in PagerDuty/go-pagerduty#218 (comment).

I'm somewhat skeptical that a breaking change is really necessary there, but even if it is, it seems like the sort of fix that would be allowed within a major version under a Go-1-style compatibility policy. So I personally don't find that example particularly compelling.

@theckman
Copy link
Contributor

theckman commented Feb 23, 2021

@bcmills The latter part of that post was showing a case where we ran into that problem that would only be a one-character change in consumer projects, but the SIV would make it so much larger. There is also this, which is a similar class of issue PagerDuty/go-pagerduty#251 which I don't believe can be easily resolved without a BC. Let's focus on this overall need / desire instead:

This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

How would you address this need in Modules?

@ulikunitz
Copy link
Contributor

Making it costly to break compatibility is a feature not a bug.

@rsc has explained it here: https://research.swtch.com/vgo-import. I don't agree with Russ on everything (#38776) but I agree with him here.

Breaking changes are bad. Who is using Perl 6? Python 3 was a pain. Successful software usually takes compatibility seriously. Windows fakes internal structures to allow major applications still to run.

Linus Torvalds wrote this:

Breaking user programs simply isn't acceptable. (…) We know that people use old binaries for years and years, and that making a new release doesn't mean that you can just throw that out. You can trust us.

I may also may remind on the Go 1 compatibility guideline was a huge factor for its success. You might want to read it again: https://golang.org/doc/go1compat

So if compatibility is such a huge factor for software success, why want to make it easy to make incompatible changes?

@ianlancetaylor ianlancetaylor added this to Incoming in Proposals (old) Feb 23, 2021
@nbys
Copy link

nbys commented Feb 23, 2021

@nbys I see you 👎 on my comment. I'm sorta gonna put you on the spot and ask if you can explain what part of that you disagree with and why you disagree with it?

I am sorry if my downvote in some way offended you.

I cannot in good faith justify forcing people to update all of their files that use our Go Module with the SIV for a one character change that they may use in one file. I'm sorta justifying the breaking change to myself because the API has never been stable due to the conflicting field names.

I could add my cents only from a library-user perspective.
I do not think of the necessity to change imports for a new major version as something bad. SIV enforces us to consider an update to the major version as a serious change.
In my opinion, it is worth to grep the project, change the import paths and actually revisit library usage.
The proposal won't change anything for me. But I believe @latest will be then a common thing in a lot of libraries. And only because it is just easier.

P.S. sorry for my English

@theckman
Copy link
Contributor

theckman commented Feb 23, 2021

@nbys absolutely no offense, a 👎 just doesn't help me challenge the ideas / opinions I have. Thank you so much for taking the time to reply so that I can understand where you're coming from. 👍

P.S. Your English was great!

@peterbourgon
Copy link

peterbourgon commented Feb 23, 2021

Making it costly to break compatibility is a feature not a bug.

The cost of breaking compatibility isn't constant, it's a function of many variables that can be different from project to project.

Breaking changes are bad.

Not universally. Breaking changes are sometimes necessary, or even good.

@deltamualpha
Copy link

I have a question about a specific part of the proposal from an application developer POV:

Application developers may consume the package via go get github.com/go-chi/chi@latest or with @v5 or @v5.0.0 and the expected import path will be "github.com/go-chi/chi", however "github.com/go-chi/chi/v5" import path would also be valid and usable.

I import and use "github.com/go-chi/chi" and get version 5. (Assume, for the sake of argument, that no other libraries I use affect which version of chi MVS resolves to.) The chi developers tag and release v6.0.0. I install a new library, and run go mod tidy. Do I now find myself depending upon chi v6?

@pkieltyka
Copy link
Author

pkieltyka commented Feb 23, 2021

@deltamualpha no, upgrading to major versions should not be implicit via go mod tidy. Once a module in go.mod is set with a specific major version (similar to how every other package manager works in other ecosystems), a developer must explicitly instruct the module system to upgrade to a new major version.

@latest would imply the current major version which would be recorded in go.mod as the latest major at the time. In order to upgrade to v6.0.0 one would do go get -u github.com/go-chi/chi@v6.0.0 or go get -u github.com/go-chi/chi/v6. However, I would argue that if one calls go get -u github.com/go-chi/chi@latest after a go.mod is set, then in this case it would also upgrade to the latest major release + version, in your example as v6.0.0.

However, the above notes are easily debatable and I don't hold any strong opinions on version management (other then of course go mod tidy should certainly not implicitly upgrade to a major version if one is already set in a go.mod).

@bcmills
Copy link
Contributor

bcmills commented Feb 23, 2021

@theckman

This decision for not using 0.x was made because they wanted to commit to not breaking the current version of the API, because PagerDuty is a critical service, while relying on the ease of major version bumps to have consumers pick up breaking changes as they get made within the library. This was the way they were going to iterate going forward, so that folks could lock into "stable" versions because they didn't want pulling in a 0.x release subtly breaking a consumer.

How would you address this need in Modules?

Honestly, I would not “[rely] on the ease of major version bumps” at all. If a downstream user needs a fix for bug, and that fix is after a breaking change, then their only options are to either backport the fix to some fork of the package (possibly a release branch), or to stop and upgrade their code before they can fix it.

Instead, for the few cases where existing, previously-supported API is unsalvageable (such as the erroneous Id field, and perhaps the erroneous Targets fields in PagerDuty/go-pagerduty#251), I would rely more heavily on deprecation and/or compatibility shims (such as {Marshal,Unmarshal}JSON methods, for those examples).

@bcmills
Copy link
Contributor

bcmills commented Feb 23, 2021

(On a bit of a side-note, someone in the community recently showed me Rich Hickey's excellent Spec-ulation talk, which goes into great detail about the various dimensions of compatibility, versioning, and namespaces. It's worth a watch!)

@D1CED
Copy link

D1CED commented Feb 23, 2021

Breaking changes are bad.

Not universally. Breaking changes are sometimes necessary, or even good.

This debate is not about ever making a breaking change but frequency and scope of them.

One camp is in favor of low frequency large breaking changes and the
other prefers more frequent and small breaking changes.

This proposal is a complaint of the second camp that they have to edit all imports on a
release of a new major version to upgrade.

Either approach has its advantages and disadvantages but the community has to
decide what to encourage/discourage.

@peterbourgon
Copy link

peterbourgon commented Feb 23, 2021

This debate is not about ever making a breaking change but frequency and scope of them.

Yes.

One camp is in favor of low frequency large breaking changes and the other prefers more frequent and small breaking changes.

No. One "camp" (i.e. Go modules) asserts that breaking changes are always very costly and should always be avoided. The other "camp" (i.e. myself and others) observes that breaking changes are not always very costly and do not necessarily need to be avoided.

Either approach has its advantages and disadvantages but the community has to decide what to encourage/discourage.

Go the language can take very opinionated stances on a wide variety of topics, because it's just one programming language among many. If any of its opinions are a problem for a potential user, that user can simply opt out and not become a Gopher. But Go modules doesn't have the same license to assert what is and isn't allowed, because it has a monopoly on package management in the Go ecosystem. If any of its opinions are a problem for a potential user, that user cannot reasonably opt out and use another tool. By and large, modules, and other mandatory tooling, has to meet users where they are, not direct them to where the authors feel they ought to be.

@ulikunitz
Copy link
Contributor

Thinking further about it: Go's target was it to make software work at scale. And at scale every breaking change creates huge costs due to the number of imports of a module. Semantic import versioning actually reduces the cost of a breaking change at scale by giving it another name. And if you publish your code on the Internet you are working at scale, if you want it or not.

@bcmills Thank you for providing the link to Rich Hickey's talk. I can add him now to the people that think compatibility is a factor for successful software.

@theckman
Copy link
Contributor

And at scale every breaking change creates huge costs due to the number of imports of a module.

@ulikunitz what if 99.99% of those who import you don't use the functionality you're breaking in the major version change, and as such it's a no-op to upgrade to your new major version. Wouldn't it be a much larger cost to force everyone to update their imports?

@peterbourgon
Copy link

Thinking further about it: Go's target was it to make software work at scale. And at scale every breaking change creates huge costs due to the number of imports of a module . . . And if you publish your code on the Internet you are working at scale, if you want it or not.

Not all Go modules get published to the public internet. Many are kept private, in corporate or other organizations.

Not all modules are widely imported. Many are used by just one, or just two, consumers — especially private modules.

Not all modules, even widely imported modules, create huge costs when they make a breaking change. Consumers pin to a version and use it without interruption, regardless of the rate of iteration on the major (or minor, or patch) version numbers.

And not all modules contain code that should work at scale, in the sense that you mean. Go is a general-purpose programming language, suitable for more than one type of user.

These are all normative claims about how the Go ecosystem should be, according to some specific (i.e. non-universal) set of assumptions. They don't describe the Go ecosystem as it actually exists.

@peterbourgon
Copy link

peterbourgon commented Feb 24, 2021

@ulikunitz what if 99.99% of those who import you don't use the functionality you're breaking in the major version change, and as such it's a no-op to upgrade to your new major version. Wouldn't it be a much larger cost to force everyone to update their imports?

Well, hopefully the goal here is to make it easier to consume major version updates, not to make it easier to violate semver 😉

@theckman
Copy link
Contributor

theckman commented Feb 24, 2021

Well, hopefully the goal here is to make it easier to consume major version updates, not to make it easier to violate semver 😉

I thought that's what I was communicating, making the cost to upgrade proportional to the change needed. 🤔

@Merovius
Copy link
Contributor

@theckman Nothing what you say is untrue. But it also doesn't contradict what I was saying.

@theckman
Copy link
Contributor

theckman commented Nov 28, 2021

I'm highlighting you once again being dismissive of someone's lived experiences, where it would have probably been better to just not reply. You keep repeating the same things while dismissing folks, even after apologizing, which is not only unhelpful but it's sorta toxic.

I've chosen to only comment here as of recently when novel things have come up, and additional specifics of the conversation we've not considered before are raised.

@Merovius
Copy link
Contributor

@theckman To be clear: I was being genuine. You responded to me, but your comment does not seem to contradict what I was saying. To me, that seems to indicate that you misread what I wrote. That seemed worth pointing out, in case you wanted to correct that.

I was (and am) being terse specifically to avoid what you say I do - keep repeating myself. I assume it's easy to read terseness as dismissiveness, but that's not my intent. And I did try to take the time to explicitly acknowledge the validity of your views in my comments, even while I was being terse.

@ocket8888
Copy link

ocket8888 commented Nov 29, 2021

If you can't trust a tool to replace an import path, you clearly can't trust your VCS, your code-review tool, the browser where you look at the review, the Go compiler or the kernel which runs your software. If that's the view you want to take, that's fine. Personally, I find that hard to relate to.

I could trust it, eventually, when it's as proven as any of those other decades-old things you mentioned. But what's even easier than that is never writing it in the first place, because the alternative is orders of magnitude lower in impact. So I know which I'll choose, if it's up to me. Especially because "replace an import path" does not update examples or documentation, which I can also skip, so I am extremely skeptical that it'd be more work to go to all this trouble and then still have manual steps.

@flibustenet
Copy link

flibustenet commented Nov 29, 2021

@theckman

The issue is that any piece of software can have flaws, as can any piece of hardware. This can happen unexpectedly when there has been no problem before

What you say justify that version in each source code is a strong feature.
If the tools to replace all the files is broken we can see it immediately, the code will not compile or go.mod will still contains the old version. Same about @ocket8888 for documentation, it's more explicit to see immediately in an example about what version it's related.
You seems to ignore all the advantages of this concept.

I also don't see what make this feature more on the side of Go team vs Go community (i believe it's even the opposite as Google use a monorepo with a fork of all dependencies). We was using gopkg.in in the community long before this. I was also using the same concept in Python since decades !

That said, i believe also that all of this was already discussed and approved by the community by large. I don't see anything new here (though my english doesn't let me understand all correctly, i works for that, sorry!).

@ianlancetaylor
Copy link
Contributor

@theckman I hear your pain but I'm not sure how to respond.

If there is one thing I have learned through repeated painful episodes in over 30 years of being an open source developer, it is that there is no good way to way to make breaking changes to a library or a tool. A corollary is that it's important to think hard before adding something to a library or tool, because there will be no good way to change it later. I long ago lost count of the number of times I caused myself severe headaches by making an ill-considered addition that I then had to support decades later in very different circumstances.

You ask "How many platform libraries that use Modules has the Go team maintained over the past 2 years on a fast moving product?" I don't know the answer, but the Go team in general uses one of three approaches.

The first is simple to describe but very hard to follow: never make a breaking change. That is our goal in the standard library and in some of the golang.org/x repos. This is a difficult and painful discipline at times, but it's the right choice for the standard library. I want to stress that this is really really hard for packages like net/http and go/types. But we do it anyhow because it's the right thing to do.

The second is to keep the project at version 0 and not make any compatibility promises. This is done in some of the other golang.org/x repos (and is the approach I usually take in my own personal projects these days). We keep compatibility if possible, but break it when it seems necessary. We implicitly expect all users to live at HEAD at all times.

The third, when we really can't keep a package going, is to introduce a new package under a new name and encourage people to migrate to a new API.

My guess is that you will not find these approaches to be satisfactory.

Adding questions to the Go user survey is a good idea. I would certainly support that.

More generally, I would summarize your comment, I hope fairly, as saying that there are problems with Go modules when library developers are forced to keep introducing breaking changes to existing libraries, because updating modules to new major versions pushes work onto their clients that is sometimes unnecessary. We know that. We knew that years ago when we adopted modules. We still know it today.

We know further that there will always be problems when using libraries and tools that make breaking changes. We certainly don't claim that we have found the perfect or final solution to that. We only claim that we have a solution that has certain characteristics that we think are good.

I could respond more granularly to the specific details written above, but of course I don't fully understand the situation, and I have the sense, perhaps unwarranted, that my comments would be dismissed. I fear that this conversation has moved away from "let's find an answer to this general problem" to "I want this specific change."

Finally, I want to emphasize's @rsc's point that having the same import path have different meanings within a program is something that we actually tried, and it had real problems in practice. The approach we are using now also has real problems in practice. But we think that the problems with the current approach are less bad. It's reasonable to disagree with that. But I don't think we're just ignoring the problems, which, again, we've heard about for years. I think we are considering them and making a reasoned choice.

@cameracker
Copy link

cameracker commented Nov 29, 2021

I long ago lost count of the number of times I caused myself severe headaches by making an ill-considered addition that I then had to support decades later in very different circumstances.

I think what people are struggling with is that in most ecosystems, a developer makes this mistake and they go figure it out with their user base. They don't wrestle with the language itself as a gate keeper.

I don't know the answer, but the Go team in general uses one of three approaches.

The other thing is that this is a false "tri"furcation. The fourth option that people reach to is to create a new major version.

In most languages, this is a happy path feature because it is understood that it has to happen periodically because people can't see the future. It is the right call for your team as a language developer to be seeking a highly disciplined level of stability. But the vast majority of developers have different priorities (often related to delivering business value in a time window) that would lead them towards incrementing a major version with an expectation that it can be released expediently.

The gift that SEMVER gave to OSS was a very simple, clear way to communicate an expectation of entropy in upgrading to a new version of a package based on the release number. SIV takes that concept and instead uses it to make upgrading major versions even scarier and more labor intensive.

Note that many of the people here complaining are folks that maintain packages. I'm reading from this thread it is effectively deliberate to dissuade package maintainers from upgrading the major version because of the spectre of major upgrade entropy (also, will point to a tense conversation where it was suggested that the azure dev team doesn't care about library stability because they're on a high major version). But in the end, SIV will never stop that behavior. People will simply work around it (and thereby cause greater instability to the ecosystem) , or complain, or move onto another ecosystem.

Are you all sure you're choosing the right tradeoffs?

@ianlancetaylor
Copy link
Contributor

The other thing is that this is a false "tri"furcation.

Sorry, I didn't mean to imply that those were the only possible approaches. I only meant that those were the approaches that I'm aware of the Go team using. There are actually quite a few other possible approaches. The Go language and tooling provide a guide path toward certain ways of working, but they don't strictly enforce them.

Are you all sure you're choosing the right tradeoffs?

Of course we're not sure. There is no certainty in this business. We're making a judgement call.

I think it's fair to observe that you are repeating the problems that have already been raised above, and that you aren't trying to grapple with the problems that I and others have mentioned. This is the kind of circular conversation that isn't going to lead to a different conclusion.

I really encourage people who want to continue discussing this to take a step back and think about the best way to handle breaking changes in dependencies, rather than focusing solely on the current requirement of recording major versions in Go modules. Because, repeating myself and thus helping to drive the circular conversation around for another cycle, simply removing major versions from import paths is not a panacea. It introduces problems that we shouldn't just ignore. Rather than ping-ponging back and forth about major versions in import paths, what is the real problem and how can we solve it?

@ocket8888
Copy link

ocket8888 commented Dec 1, 2021

I think what people are struggling with is that in most ecosystems, a developer makes this mistake and they go figure it out with their user base. They don't wrestle with the language itself as a gate keeper.

This is my exact concern. Go has made a decision for all projects that use their language, a decision about versioning and releases. These things are not in the domain of things a language controls. I've tried to show how the versioning system poses mechanical problems for me, but those aren't the biggest problems, in my opinion. There are other things I don't like about Go, like a lack of conditional compiling, or no function overloads. Those things, though, are all aspects of a programming language - and have workarounds (as of generics, anyway). It's totally fine for Go as a language to not support some language feature, or to enforce strict standards on what constitutes a valid parse tree (e.g. unused variables causes compilation failure), even if I don't like it.

It is morally incorrect to impose project direction decisions on projects you don't control. People I have never met are telling me how to release my software. If I want to make breaking changes, it is not the place of the Go language to tell me I can't.

And not that it has any bearing on the morality of SIV, but like I mentioned ATC is not really a library. It's an API, and we semantically version our API. Our API clients never experience breaking changes according to that semantic versioning. If you import the APIv2 client, then you will have no breaking changes until you decide to upgrade to the APIv3 client - which is a different import path. The structures for our API live in a shared library, and the structures for each major version have V{{N}} appended where N is the API major version number. These do not experience breaking changes, until you decide to upgrade to a newer API version, whereupon the struct name changes.

So not only do we use semantic versioning, but we even used versioned import paths for our client. None of that is good enough for Go. The way we communicate breaking changes is somehow being dictated by the programming language we chose to implement it in - and that's frankly ridiculous. It's incompatible with converting an existing project and process into Go, and to make matters worse, it's unnecessary, because it would be a non-breaking change to the tools to simply allow project maintainers to maintain their own projects how they see fit. Even if you don't like it, I cannot comprehend not allowing it; it's simply not your call.

@Merovius
Copy link
Contributor

Merovius commented Dec 1, 2021

This is my exact concern. Go has made a decision for all projects that use their language, a decision about versioning and releases. These things are not in the domain of things a language controls.

This seems incorrect to me. Either a) Go did not use the language to make that decision, or b) most languages make these decisions. The Go language does not know about modules - the go tool does. And most languages come with package management tools, which make exactly the same kinds of decisions using exactly the same kinds of mechanisms.

@theckman
Copy link
Contributor

theckman commented Dec 1, 2021

And most languages come with package management tools, which make exactly the same kinds of decisions using exactly the same kinds of mechanisms.

@Merovius to make a statement like this you need to show us another ecosystem that's made similar decisions as modules, without it resulting in the problems we are facing here.

@ocket8888
Copy link

The Go tool compiles my code by linking imported code using module definitions. The Go grammar doesn't know about modules, but the language does.

@Merovius
Copy link
Contributor

Merovius commented Dec 1, 2021

The Go tool compiles my code by linking imported code using module definitions. The Go grammar doesn't know about modules, but the language does.

Sure. Fair enough. But the same is true for Cargo, NPM, gems, pip…

Again, the problem is calling it a "language decision" for Go, but a "tooling decision" for other languages. It's either a language decision in both cases, or it's a tooling decision in both cases. I'd be on board with either definition.

@theckman
Copy link
Contributor

theckman commented Dec 1, 2021

@Merovius please show us in those languages what constructs of the package manager end up in the source code of the project. Because if they don't, it's not the same at all.

Edit: the fact you have to import a package by it's name doesn't count. If it's name + version identifier, or other package metadata, that would be a better analogue to Go.

@AlekSi
Copy link
Contributor

AlekSi commented Dec 1, 2021

Deno uses the full version in import statement: https://deno.land/manual@v1.11.5/examples/import_export#remote-import

(but I'm pretty sure that example will can be twisted one way or another to support already-established opinions)

@theckman
Copy link
Contributor

theckman commented Dec 1, 2021

@AlekSi I could maybe argue they observed the problems we're experiencing with Modules, and chose to require explicit versions on all imports, versus implicit (major) versions and having it not be present for v0 an v1. I'm not sure Deno is mature enough for patterns to emerge around problems that may create, but I appreciate you highlighting it as it means it's a space we can watch from afar.

@ocket8888
Copy link

ocket8888 commented Dec 1, 2021

Sure. Fair enough. But the same is true for Cargo, NPM, gems, pip…

Again, the problem is calling it a "language decision" for Go, but a "tooling decision" for other languages. It's either a language decision in both cases, or it's a tooling decision in both cases. I'd be on board with either definition.

NPM and pip are not the compilers for Javascript or Python code. I don't have experience with Ruby or Rust, but I'd bet it's the same. Go code fails to compile for other people if I don't change my module path. That is not the same thing. go build is the reference compiler implementation for Go. go build is a part of the language. This is a problem with the linking step in compiling Go language source code, not a tooling issue. This isn't the equivalent of pip install, we're talking about the import keyword, if you want to use a Python analogy - although that's a rough analogy because Python as a language doesn't use SIV and a Python module is very different from a Go module not only because of that.

@mvdan
Copy link
Member

mvdan commented Dec 1, 2021

Could I encourage everyone to take a step back from this thread? It's extremely long and has been closed for some time.

Experience reports are very much welcome, and the wiki is a far better medium so they don't get buried underneath hundreds of comments.

If groups of users are encountering significant problems with modules, and they feel like a solution isn't being prioritized, collecting experience reports will be significantly more effective in terms of capturing evidence to make the right design decisions for the years to come.

@theckman
Copy link
Contributor

theckman commented Dec 1, 2021

@mvdan your comment doesn't match my and others' experiences. In the past when we brought experience reports, folks like @bcmills would dismiss them similar to Steve Jobs's comments during the Antennagate situation years ago. Instead of using that experience report to try to inform their opinion on the topic, they'd spend the entire time telling us how we are wrong.

So if you're going to advocate for us to pursue a process that was unhealthy in the past, can you highlight what is in place now to prevent those types of interactions from people in positions of power?

@mvdan
Copy link
Member

mvdan commented Dec 1, 2021

What experience reports did you submit? As far as I can tell, just one has been added to the wiki's Modules section since mid-2019.

@ocket8888
Copy link

Could I encourage everyone to take a step back from this thread? It's extremely long and has been closed for some time.
Experience reports are very much welcome, and the wiki is a far better medium so they don't get buried underneath hundreds of comments.

Yeah, for sure. Like I mentioned in my first comment here, I'm happy to have this conversation wherever is appropriate.

@Merovius
Copy link
Contributor

Merovius commented Dec 1, 2021

@ocket8888 You might be unaware, but the go CLI command (commonly called "the go tool") is not the Go compiler either. It is a build-tool, capable of invoking either gc or gccgo, which are both Go compilers. There are also several other Go compilers (I know of TinyGo, llgo and GopherJS) and other build tools (examples I know of include dep and bazel). Every build tool comes with its own convention of how it maps an import path (which is the Go language construct to express a dependency) to a set of source files constituting that package (possibly including version resolution and remote discovery).

So, your comment is just not true. The go tool is exactly analogous to pip and NPM. And gc/gccgo/llgo/… are analogous to CPython/Jython/… and V8/SpiderMonkey/…. And an import declaration in Go is analogous to an import statement in Python or JS.

It is also not true that the language import construct does not usually reflect decisions that the packaging tool made. A very visible example is Java, where an import declaration in the language is simply an identifier, but conventionally (similarly to Go) it is constructed from a fully-qualified domain name and used to look up the actual file containing the relevant code in the filesystem.

More importantly, even a language like Python does reflect the decisions made by the packaging tool in its source. Specifically, SIV is the direct result of two decisions made by the go tool:

  1. It should be possible to import and use two major versions of the same package.
  2. The import path in source should map injectively unto the actual packages in the build (i.e. if two different packages use the same import path, they should also refer to the same code).

Now, pip also decided on these two questions. It made a different decision (AFAIK not to allow two major versions) from Go. But it made a decision. And nothing in the language would prevent a packaging tool for Python to make different decisions about this - and it would still be an implementation of the same language. For example, Jython (a different Python runtime) still implements Python, but it also has to adhere to the conventions set by the JVM for import statements, if it wants to interact with Java code.

Therefore, the fact that Python code (intended to be consumed by common package managers) does not mention the major version of the imported package is a reflection of the decisions its package managers made.

Which is why I @theckman's limitation

the fact you have to import a package by it's name doesn't count. If it's name + version identifier, or other package metadata, that would be a better analogue to Go.

is not a demand to "show another language's package manager which had to make the same decision", but a demand to "show another language's package manager which came to the same conclusions as Go". I can't answer that demand, because I don't know other languages well enough - for all I know, the tradeoff Go chose is unique. But that's immaterial. My problem isn't with the claim that "Go's specific chosen decision is bad", my problem is with the claim that "even making this decision is morally wrong". Every language made this decision, they might've just landed someplace else. Which is fine.

@willow997
Copy link

willow997 commented Dec 1, 2021

So if you're going to advocate for us to pursue a process that was unhealthy in the past, can you highlight what is in place now to prevent those types of interactions from people in positions of power?

@theckman FWIW I've followed the design space around go modules since the early days of godep, gb, glide, dep, vgo, and finally what we have today. I've found your contributions this debate consistently unhealthy with appeals to representing the community, and making things unnecessarily personal with the go team. I've seen similar behavior with others who had issues around the dep/vgo debates. I'd ask that you not let past issues stemming from dep drama color all future discussions around modules. It makes it hard for others to have productive conversations.

I personally like what modules have given us and agree with nearly all design decisions. I've even seen and participated in the go team listening to and changing their minds on some module tooling issues. However after using modules and seeing others use it I've come to the conclusion that I would have rather had a world without SIV in favor of a culture of changing import paths for some projects. That ship has sailed though and I see UX advantages to both for certain use cases. Instead, I am very interested in future discussions that may consider tooling improvements to ease the pain points of SIV, but I've stayed away because often these discussions seem dominated by those with personal issues with the go team and the entirety of go modules. That doesn't help the community potentially improve the tools or go.

@lpar
Copy link

lpar commented Dec 2, 2021

FWIW, Ruby allows an explicit version of a module (RubyGem) to be required (imported), e.g.

gem 'hubspot-api-client', '=11.1.1'
require 'hubspot-api-client'

There are operators for <, >, <=, >=, ~>, =. Or you can just use the require line and get the most recent version that's installed.

So in Ruby, the Go-like behavior of requiring specifically version 2.x of a library and not allowing use of newer versions without changing all your imports is supported. In practice, I've never seen anyone choose to do that. However, if someone wanted to do a review of major Ruby projects, maybe it's more common than I think?

@ianlancetaylor
Copy link
Contributor

@ocket8888

Go has made a decision for all projects that use their language, a decision about versioning and releases. These things are not in the domain of things a language controls.

The Go project has a different opinion: they are explicitly in the set of things that Go has an opinion on. As @rsc said above: "The fact is that Go is an entire programming environment, not just a language, and handling dependencies well is one of the core reasons we created Go."

That said, @Merovius is correct in pointing out that go build is not a compiler. People do use the Go language with other build tools, most obviously, as mentioned, Bazel (https://bazel.build/). People who use Bazel to build Go programs typically use a completely different approach to dependency management.

It is morally incorrect to impose project direction decisions on projects you don't control.

I'm sorry that the go tool is making what seems to you to be an immoral imposition. Go is definitely an opinionated language, but these are technical choices that to me, and I think to the other people working on Go, do not rise to the level of morality. Given your feelings about this it is possible that Go (or at least the go tool) is not the right choice for you.

So not only do we use semantic versioning, but we even used versioned import paths for our client.

I"m sure I'm missing something, but I don't understand why you are having trouble. The Go way would be for your stable API packages to use version 1.0, 1.1, 1.2, etc. Then there won't be any major version change, and no need for clients to change their import paths until they are ready to change to a newer API.

Perhaps this is what you meant when you said earlier that one of your options is "Lie about our module version." Yes, that may be an option. Only it's not a lie. A new major version implies a breaking change, and you don't have breaking changes. So you should keep the same major version. At least, that is the approach that the go tool expects.

@ocket8888
Copy link

ocket8888 commented Dec 3, 2021

The Go way would be for your stable API packages to use version 1.0, 1.1, 1.2, etc. Then there won't be any major version change, and no need for clients to change their import paths until they are ready to change to a newer API.

This is only possible by either never making breaking changes, or lying to users. Both are unacceptable, because we need to move forward, and we need to communicate those changes.

I think you're not understanding something about the project's layout (apache/trafficcontrol if you're interested) Changes to the API - and thus the client - are not the only reasons we'd release a new version. ATC is a collection of software products to be deployed. Some are written in Java, some are front-end webapps, several are written in Go. We make new major releases of our software whenever any breaking change is being released, to any of them. It makes compatibility charts simple. We are currently on version 6.0.1. Even if it were possible to never make breaking changes - it isn't, in a real-world application used at scale, IMO - it's far too late for that. We released 2.0 on July 6th of 2017, more than a year before experimental module support was added to Go 1.11.

So while "lie" may bring with it a connotation of maliciousness, I'm sure you'll agree that e.g. having a v1.6.1 tag that actually points to software at version 6.1.0 is at best confusing. Not only confusing, actually, but in fact that loses information, since now we cannot communicate patch releases using Go modules. To make matters worse, we already have a semantically versioned API separate from our software versions. For example, the Traffic Ops component of ATC at version 6.0.1 supports API versions 2.0, 3.0, and 3.1 (and serves an unstable/unreleased 4.0 API). So your suggestion means that people using our official Go client need to sort not merely two, but three separate versions - a tag v1.6.1 exists that they import with Go that gets them the client released at version 6.1.0 that works with the presumably released 4.0 API. It's all just kind of a lot.

Given your feelings about this it is possible that Go (or at least the go tool) is not the right choice for you.

For personal projects, sure, but we just finished a two-year effort to rewrite our Traffic Ops component from Perl to Go at the beginning of this year. That component alone represents 628 non-dependency source files. Throughout the project as a whole, we use 250,006 lines of Go source (not stripping blank lines or comments) across 1141 files (again not including vendored dependencies). If that ever changes, it likely won't be in this decade.

Besides, Go is fine, it's go I could do without, apparently. I once tried to compile one of our components with gccgo and couldn't get it to work. Gave up after a couple days. It's good to know there are alternatives, though.

It sounds like the only solution is to not use go to build, then, which is an absolute shame, not simply because that is the de-facto standard tool for working with Go and our users were hoping we could get the breaking change that it no longer works with our software resolved somehow, but also because we can't use go doc anymore, our pkg.go.dev docs will never show the correct version and/or information (not that they ever did, but it would've been nice), IDE integrations and language servers would be broken etc. etc.

We could still choose to abandon support for go, but I think we all know that's a terrible idea. When this issue first cropped up, we had a discussion with someone from the Go project, and when we laid out our options he had this to say in response to the proposal to simply not use modules:

Please no; this would be a regression.

So I was hopeful that there was some alternative. I don't really want to call him out by name to prove he's associated with the Go team or whatever, he's been nothing but helpful and generous with his time.

If our only path forward really is to require that anyone importing our client use something besides the go tool to do it, then it really seems like go has made a breaking change. People continue to repeat the perceived merits of SIV, and I'm not so very interested in disagreeing, I'm just baffled at the refusal to make the non-breaking change to simply allow things to work.

...handling dependencies well is one of the core reasons we created Go.

we'll have to disagree about the definition of "well" in this context :P

I responded because you @'d me directly, but I have been asked to take this conversation elsewhere, so I'm sorry if it seems rude - or possibly cowardly - but I won't be replying here further, even if someone else does @ me, now that I'm making that clear. Thank you to (mostly) everyone for your time, patience, and contributions during this necromancy.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
No open projects
Development

No branches or pull requests