New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
cmd/go: go mod tidy
ignores go.work
file
#50750
Comments
Tidiness is a property of an individual module, not a workspace: if a module is tidy, then a downstream consumer of the module knows which versions to use for every dependency of every package in that module. If you don't particularly care about downstream consumers having a package that is provided by the workspace, you can use Otherwise, you either need to publish the workspace dependencies before running |
@matloob, for Go 1.19 I wonder if we should augment the |
go mod tidy
ignores go.work
` filego mod tidy
ignores go.work
` file
go mod tidy
ignores go.work
` filego mod tidy
ignores go.work
file
Currently with reproducing repository (https://github.com/bozaro/go-work-play/tree/go-mod-tidy). I can run:
And But for This behaviour looks like inconsistent: I expects that |
What valid use cases are there for |
This is also happening for I agree with @liadmord, the only benefit I see get from the |
I agree with @bozaro @liadmord and @AlmogBaku I have a monorepo private project where I have multiple modules, and on each one I need to insert a lot of |
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
This comment was marked as duplicate.
Encountering the same issue with Furthermore, some other
Expected behaviour: The |
My expected functionality is that a If I don't commit my Once I must make an edit to In other words, if I add this line to
Then |
This is a very big issue for me in the case of creating Docker containers :( I cant use Golang is considered the language for microservices and In the current state of the
I expect 4 things from the
So now, the only thing why I'm using |
Signed-off-by: Valery Piashchynski <piashchynski.valery@gmail.com>
I've faced this issue yesterday but my case is a bit different.. My The solution is simply to move the |
Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered |
@bcmills I'm not sure what the intended workflow is, but I think the feedback here is that the DX is really not intuitive. |
I wish there were |
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
This comment was marked as off-topic.
However that replace directive seems to be ignored by
|
@matloob can we not XY-problem this and take as given that it is a valid use case to have a client and a server be in different modules, especially considering it is commonplace for them to be in entirely different repos? Otherwise this turns into a tangent of "why are monorepos, especially ones where packages within can have dependencies on other packages' source, beneficial", which is deeper topic than needs to be explored here, and it can just as easily be taken a given that a non-trivial number of organizations (and therefore go toolchain users) have decided this has some good answers. For further clarification, I used the term "package" above in the general sense, not the golang-specific sense; pretend I said "module" if you prefer. |
@matloob: i'm in the same situation as @mdepot. In many cases, the client is free / open-source while the server is proprietary / closed-source. There are also cases where one side has far more dependencies than the other (i.e. why force people to pull a fat Electron client when they only want a server side that is strictly files & sockets?). |
We also use a similar structure for the above mentioned reason. When running Go tidy from the root, it took us by surprise that it ignored the go.work. A quick fix was running:
But that is just a temporary solution, for a missing feature. |
@dkrieger, generally we expect “dependencies on other packages' source” to mean something that can be fetched as a module, from a (public or private) GOPROXY or version-control host. It isn't obvious why one would have dependencies that can't at least be redirected for |
That was the motivation for adding module graph pruning in Go 1.17. Are there cases of this sort where graph pruning is ineffective? If so, can you give more detail about the concrete problems (ideally as a separate issue)? |
@bcmills The obvious case is when you have a monorepo with multiple go modules, orchestrated with that other Google tool bazel/blaze. Being able to publish updates to two or more modules in your monorepo in the same commit is high on the list of benefits of a monorepo. |
@dkrieger, can you give some more detail about why the monorepo is structured that way? (We're trying to better understand the underlying use-cases so that we can address them holistically.) We're familiar with https://github.com/googleapis/google-cloud-go, which IIUC uses multiple modules so that they can tag its various APIs at different levels of stability (for example, have some APIs stabilized at v1 while others are still at v0). But the discussion on this issue has been mostly focused on dependencies that don't have any versions published at all, which would imply that that's not the same underlying motivation. |
@bcmills, "pull" as in git/hg of a repo with large assets (like a GUI with many rasters/videos) when I don't need it. No complaints on compile time or binary sizes. |
Hmm, here's a thought. For #50603, we're going to need the ability to inspect the local repo containing a given repository to determine its version information. In theory, But there is a bit of a chicken-and-egg problem: the version of a checked-out repo in the workspace depends on its commit hash, which depends on the contents of its files. If there is a cycle in the requirement graph (which is normally allowed), then
|
@bcmills let me start with a language agnostic description of why we would structure a monorepo in this way, then I'm happy to field any go-specific follow-up questions you might have. At our company (and I'd suggest most companies), most of our code is not public. Our deployable artifacts are mostly containers that we deploy to kubernetes. For some given deployable slice of the codebase, we may deploy it as one or more services, and a given chunk of code may appear in one or more containers -- in a common scenario where it appears in multiple containers, we may have a fatter container that aggregates multiple domains (let's say each domain is represented by one grpc service, and we pack those into single networked service, i.e. many:1 domain:service) and a smaller container that is a microservice (i.e. 1:1 domain:service). This enables sophisticated traffic shaping when both are deployed, but in more simpler scenarios it means we can periodically move predictable-traffic and/or high-reliability domains into the aggregate service, and move bursty and/or low-reliability domains into their own microservices -- basically, we preserve domain boundaries and the flexibility of deploying as a microservice or in an aggregate/monolith as we see fit, balancing performance, complexity, efficiency, and reliability over time. At lower levels, we may have various utility libraries that get used by many deployable slices of the codebase. Now, for all that private code, we don't really care about publishing these for consumption from other repos, whether inside or outside the organization. As such, we don't have to worry about versioning in the traditional sense -- any commit on the trunk branch (or, by convention, any merge commit) can be built and deployed, and the commit hash is the "version" for everything. Because everything we do is in the monorepo, we not only know every consumer for X service, but they share the same build and IDE context. We can make breaking changes to a service and update every consumer in a single commit if we want to (in practice we'd make a backwards compatible change followed by cleaning up the deprecated API when talking about services, but in the case of libraries we can do it all in one shot). Still, we want strong boundaries between different parts of our codebase, and the universally strongest boundary is to structure it as the unit a given language uses for external dependencies -- in node, this is a package; in go, this is a module. We don't actually want to put package registries in the middle of this though. When I look at my string util package ("module" in go), I want to be able to look at the go.mod file and see just the dependencies for that, not polluted with a single line that has to do with some consumer of the string util go module, or some other completely unrelated go module. If it doesn't depend on any other go modules in the monorepo, it could be built with just the standard go toolchain. For higher level go modules in the monorepo (e.g. a networked service), if I don't want them published for consumption outside the monorepo, they will have dependencies on external libs and some other internal modules in the monorepo. I'd like to be able to |
@dkrieger Could you speak about how you use Bazel together with modules? It was my understanding that Bazel didn't support modules. Is that incorrect? Why do you need modules if you're using Bazel to do your builds? Do you have a Bazel WORKSPACE for each of your modules? My understanding is that the workspace in Bazel is what maps closest to a Go module. |
@matloob Bazel with rules_go and gazelle lets you configure gazelle to source your workspace-level go dependencies from the union of every go.mod in your bazel workspace. A bazel workspace maps to a go.work, if you are using go modules and workspace. The only pain point is the missing ability to have go mod tidy manage a given go.mod for you that uses your unpublished/unversioned sources for other modules in the same go/bazel workspace |
@dkrieger I didn't realize that Bazel supported that. I want to try to step back a bit. The reason I'm so hesitant about supporting this is that we don't want the tooling to encourage folks to use multiple modules when they can work with a single one. The motivation that I understand for having a multi module monorepo is that the modules are distributed separately from each other. That is: the modules appear in a monorepo, but they are also dependencies of other external modules. For those cases go mod tidy should work fine because each module is able to stand on its own as a dependency of an external project. So I want to understand your use case better. I see that you mentioned the string util libraries. Are those depended on by other modules outside of your monorepo? Are they built separately from the rest of your monorepo? |
I have a couple responses to this:
For all intents and purposes, I'd say that the use case I'm describing does involve distributing modules separately from one another; whether they're distributed in the conventional go way or not, whether they're published to a public registry (including public github repo) or not -- these are downstream decisions. In all cases, they're possibly (and in practice, often) developed in lock step, and contracts can be changed in backwards incompatible ways in a single atomic commit. Again, the main effect is eliminating the toil of traversing your dependency graph and the logical change you're wanting to make becoming distributed over the dimension of git-commit-time The result is much cleaner than having a huge module with packages that have no logical relationship with one another. Every module I create tends to consist of several packages, and those packages all pertain to the logic of that module. The 2nd level of ordering that modules provide is incredibly useful, as it (1) makes it easier to understand the transitive closure of dependencies of any piece of code without noise and (2) makes it easier to understand the interrelated packages that exist in that module. This is just as true whether or not I publish my modules in a way that they can be retrieved via In practice, I'd only invest in versioning/publishing for consumption in external repos if I want legacy pre-monorepo code to use them. I don't want to extract a network of packages from a monolithic module when that determination is made, I want to make small changes to my distribution logic (and introduce versioning) |
When downloading a module the entire repo is checked out. So if you have a single go.mod for a monorepo you easily run into an issue where you can hit the limits of size of repo (I can't remember what the limits are but it's hardcoded into the go command). Using submodules does inform git to ignore directories. I am specifically running into this problem with trying to add new modules to an existing repo. New module is created, needs to be referenced by other modules in the monorepo but Replace works here, but sorta defeats the purpose of |
Yep, that is how I had to work around it. I don't think it has anything to do with limits on the size of a repo though. |
What i mean is that I am using (or rather want to us) multiple modules to get around the size of my monorepo vs just having a single There are two problems with large monorepos and a single For all download type it looks like it defaults to 10<<20 (or around 524MB) which is a lot for a repo, but we use git-lfs which is resolved here and includes any binaries you may want to stuff into your monorepo: https://github.com/golang/go/blob/master/src/cmd/go/internal/modfetch/codehost/codehost.go#L35 Submodules is a great way to solve this, but because of resolving versions and the chicken-and-egg problem they are hard to work with and hard for my coworkers to stomach. |
An important underlying theme here is that monorepos are not monolithic codebases, nor are they distributed eventually consistent codebases where you necessarily version and publish each part to some registry. They are no less modular than multi-repo/polyrepo -- to the contrary, they promote mindful modularity by reducing the incidental disincentives to modular design. That they don't encourage/require long-lived support branches in the form of versioning in many circumstances is frankly unrelated to modularity. If we can agree that this is at least a defensible position, let's try to shift the focus to how we can decouple go module tooling behavior from opinions for/against monorepo adoption without breaking userland. |
"go mod" commands are scoped to modules when making changes; there is no workspace in go mod sub-commands. This is why "go work vendor" is different from "go mod vendor". So "go mod tidy" should not know about go.work any more than the other commands. (Some of the read-only commands like "go mod graph" are workspace-aware, but that's less problematic than read-write commands.) It seems like "go work sync" is what we should be focusing on. What does it do or not do that is inappropriate? |
@rsc Hi Ross, thanks for looking into this.
ScenarioImagine this scenario, you're planning on creating a repo with the following:
ProblemSo now you learn about "go workspaces" that seem like they could help with this and you want to quickly try it out. You make the above file tree, then do:
To workaround you can do:
But that's obviously silly because you might as well not use workspaces at that point. Possible Solution 1:
|
Respectfully, this is not a solution, it's a temporary mitigation at best. Not supporting atomic updates spanning multiple workspace modules defies the purpose of a workspace in the general sense of the term, as it is observed across languages and monorepo build tools. Whether we get Updating go.mod files appears to be within the purview of |
I don't know whether it is related but one issue that I have encountered is, while building something similar to gonew, when downloading a module in a workspace, the go.work file needed to be updated manually. Perhaps that go work sync could also work bottom up (via a flag?) to add this module/directory to the module list in the go.work file? If that makes sense. |
Many use cases were described by participants here. Mine is that I do not want to publish dummy stub modules before I can start to work. I can buy rsc explanation why
that does not try to reach to unpublished modules. Ie. |
@dpifke Thanks, that's a useful perspective. We don't want our tools confusing users with the instructions they give. I'm interested if solving the error messages is the primary reason most people want 'tidy' for these use cases. @brianbraunstein I don't think 'tidyness' is the right concept here. I don't think we should call a @ohir I don't know if @dkrieger I don't think we should frame this discussion around what the concept of a "Workspace" is in other language ecosystems. We want to build what make the most sense for the Go module system. |
It still reaches to the net even if it has everything replaced or internal. It should not. Repro tree attached (usage: |
When I disable the network your repro case doesn't produce an error when the two replaces in the go.work are there. But if I remove them I do get an error. Is that what you're seeing too? If not, what version of Go are you running? |
There is no problem with Since This is a wider problem: The only working solution for now is to replenish all
Of course Summary: |
I don't understand why this discussion has continued for this long, honestly, and yet there's no progress to be seen. Your summary seems to sum it up perfectly and I think it's the reason I subscribed to this issue a long time ago: My expectation was to use |
@matloob After adding Filtered excerpts follows: [cmd/smth imports [other module - imported by cmd/smth but NOT importing
To sort this mess out an explicit replace had to be added to the (cmd's) go.mod I hear "its a gopls issue". No, it is not just I would like my "Summary" warning to be included in the docs of the workspaces for the time being - to save headaches for others trying to add some isolated module to their workspace. Thanks. |
What version of Go are you using (
go version
)?What operating system and processor architecture are you using (
go env
)?go env
OutputWhat did you do?
Minimal reproducing repository: https://github.com/bozaro/go-work-play/tree/go-mod-tidy
Full script:
What did you expect to see?
I expect
go.mod
andgo.sum
updated with current working copy state.What did you see instead?
I see that
go mod tidy
try to get modules forshared
go modules from repository ignoringgo.work
content.The text was updated successfully, but these errors were encountered: