Tidiness is a property of an individual module, not a workspace: if a module is tidy, then a downstream consumer of the module knows which versions to use for every dependency of every package in that module.

If you don't particularly care about downstream consumers having a package that is provided by the workspace, you can use go mod tidy -e to ignore the error from the missing package.

Otherwise, you either need to publish the workspace dependencies before running go mod tidy (so that they have well-defined upstream versions), or tell the go command what those versions will be (using a replace and require directive in the individual module).

@matloob, for Go 1.19 I wonder if we should augment the go.work file to be able to declare explicit intended versions for the modules in the workspace. Then go mod tidy could use those versions instead of looking upstream. 🤔

Currently with reproducing repository (https://github.com/bozaro/go-work-play/tree/go-mod-tidy).

I can run:

cd tools/hello
go run .

And go run works like workspace go modules declared as replace in current tools/go.mod.

But for go mod tidy I need add replace manually.

This behaviour looks like inconsistent: I expects that go run and go mod tidy would use same dependencies.

What valid use cases are there for go.work if we are still required to fully implement the go.mod in order to work on multi module repo?
@bcmills

This is also happening for go get ./...

I agree with @liadmord, the only benefit I see get from the go.work is a better "understanding" of my IDE
Or - maybe I'm doing something wrong here?

I agree with @bozaro @liadmord and @AlmogBaku

I have a monorepo private project where I have multiple modules, and on each one I need to insert a lot of replace statements for each direct and indirect dependencies. With go workspaces I really want to be possible to run go mod download or go mod tidy without the needed to write a replace statement in every go.mod file in my workspace.

Encountering the same issue with go1.18.2.

Furthermore, some other go mod commands also failed. See sample repo: https://github.com/maxsxu/go-workspace

• go mod vendor

➜  server   go mod vendor 
atmax.io/go-workspace/server imports
        atmax.io/go-workspace/commons/pkg: no required module provides package atmax.io/go-workspace/commons/pkg; to add it:
        go get atmax.io/go-workspace/commons/pkg

• go mod verify

➜  server   go mod verify 
atmax.io/go-workspace/server : missing ziphash: open hash: no such file or directory

Expected behaviour: The go mod commands should not try to find modules which have been declared in go.work. which I think will make go workspace more meaningful.

My expected functionality is that a go mod tidy on a go.work enabled workspace should feel like a bunch of phantom replace statements inside the go.mod without editing the file. If the module is at github.com/Gbps/mymodule and I tell my go.work that the module can be found in /path/to/mymodule, then go mod tidy should pull the version from my local repo instead of going to github at all.

If I don't commit my go.work, then anyone who uses my go.mod afterwards will have to go to GitHub, which is correct! Now I can edit my local copy and test changes, but when I'm ready to push, I can tidy the correct version into the go.mod and push both the mymodule local dev repo and my current project. This makes sense!

Once I must make an edit to go.mod to include a local path string for any feature of the go mod command line to work, all the purpose of go.work is lost on me. I should never have to taint go.mod with local paths, go mod tidy can find what commit I'm working with locally through go.work and update it accordingly. It should be no different than if I pushed a commit to github and ran go mod tidy to pull the newer version into my go.mod.

In other words, if I add this line to go.mod:

replace github.com/Gbps/mymodule => /path/to/mymodule/

Then go mod tidy works as expected. If I have use /path/to/mymodule/ in my go.work, it should accomplish the same thing. The benefit being I don't have to taint go.mod with my local paths, as that is going to need to be committed to remote whereas go.work isn't.

This is a very big issue for me in the case of creating Docker containers :( I cant use go mod tidy and copy the dependency graph to the Dockerfile. So every time I build a container I have to download downstream dependencies instead of doing go mod download.

Golang is considered the language for microservices and In the current state of the go work it's very hard to work with them in monorepo. Creating multiple-container environment is incredibly hard :(. Basically, every time I hit docker-build it has to download downstream dependencies. Currently, I have two ways to fix this:

1. Push packages to the repo. But if I have ~30 microservices and most of them are 1 file program - it's a very poor experience.
2. replace and require create too many relative dependencies and it's very hard to maintain.

I expect 4 things from the golang workspaces:

1. go mod to know about other modules and don't try to download the local package from the internet. Maybe traverse up in the folder hierarchy and check if go.work exists?
2. you can create a clear dependency graph so it can be used for caching in Docker containers
3. functions like go work tidy and go work download, because now I have to enter every package and type go mod commands. I just want to tidy and download everything from one place, and because we already have a path to the modules in go.work file - it should be trivial.
4. functionality to reduce the number of similar dependencies (download one specific version and force to use it in every module in workspace). This was done by for example turborepo, lerna, and other monorepo management tools.

So now, the only thing why I'm using go work is to have a better codding experience in VsCode. Most of the negative comments that I hear from colleagues about golang are about poor modules/workspaces management. I think, go work is a very important feature to have and it's a very important direction to go.

I've faced this issue yesterday but my case is a bit different.. My replace statements are in the go.work file, which seemed to solve go run errors, but when attempting go mod tidy on any of the modules, it is indeed ignored.

The solution is simply to move the replace statements to the go.mod files as explained by Gbps, but keeping track of what package uses what internal package is a bit of a hassle. Is this considered a bug also?

go mod tidy is intended to update the module's self-contained dependencies. It doesn't use the workspace because in general one may work on multiple independently-maintained modules in the same workspace, and if you're preparing an upstream commit you definitely don't want that commit to rely on unpublished local modifications.

go work sync is intended to update the modules within a workspace to reflect the dependencies selected in that workspace. That is probably what you want if you are working on a set of modules that are all maintained as a single unit.

Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync for your use-case? If so, what does go work sync do (or not do) that makes it unsuitable for your use-case?

@bcmills I'm not sure what the intended workflow is, but I think the feedback here is that the DX is really not intuitive.

Folks who are commenting here (and please bear in mind https://go.dev/wiki/NoPlusOne!) — have you considered go work sync for your use-case? If so, what does go work sync do (or not do) that makes it unsuitable for your use-case?

I wish there were go work tidy or something similar that would do the same as go mod tidy for each module in the workspace. Or maybe go work sync could have a flag for that, or even do it by default. This way one could easily manage a single repository with many modules, keeping everything in sync and tidy :) without having to switch to every single module directory to keep it tidy.

go work sync does not work for me in that scenario either unfortunately. I have a go.work for a monorepo containing yet to be published packages (the domain also does not exist yet). e.g.:

use (
    ./foo
    ./bar
    ./fubar
)

replace not-yet-created.com/foo/foo v0.0.0-00010101000000-000000000000 => ./foo

However that replace directive seems to be ignored by go work sync

$ go work sync
go: not-yet-created.com/foo/foo@v0.0.0-00010101000000-000000000000: unrecognized import path "not-yet-created.com/foo/foo": https fetch: Get "https://not-yet-created/foo/foo?go-get=1": dial tcp: lookup not-yet-created.com on 127.0.0.53:53: server misbehaving

Since this thread seems to summarize many different opinions and ideas as to what to change for the go.work behaviour let me add my two things I would love to have for "early development" usability.

1 dependency management for "initial setup"

I personally hate it that if I work on several new submodules i would need to release an initial version first for all of them to be able to reference them correctly on a main go.mod file. Otherwise the IDE always complains...

why not allow to reference inside go.mod something like a keyword to emphasize that this is handled by go.work file and is not maintained currently (yet) against a stable version.

something linke:

require https://github.com/google/uuid vWorkspace

linters can also work with such a keyword showing the warning for this... builds could fail initially when keyword is found. Same for PR checks.
And for initial setup you can finish a submodule with main module tests before pushing or releasing it.

Currently my workaround is:
to push it to a branch and use the branch as a version this could help others as well currently maybe.

With this keyword also go mod tidy would just work. Because the expectation of "this is not a final version" is fulfilled and IDE compiler etc still works without extra effort of manual branch versioning.

2 extra work for submodule tidyness

The second point I read here somewhere in the chat already and it is the need to "cd" into all directories to "go mod tidy" all submodules.

Workaround is to edit the make file:

tidy: ## Tidy go modules
	go work sync #sync root
	cd apis/A/v1alpha1 && go mod tidy # tidy all submodules
	cd apis/B/v1alpha1 && go mod tidy
        ...
        go mod tidy # tidy main module

So here I would love an option to do something like:

go mod tidy --recursive #(and if there is no main module it should still look into subfolders...)

I think the "featureset" that is available is usable but with manual effort which is confusing as most of the things where also possible with manual effort without the go.work file and this spikes confusion.

conclusion:

IMHO this two small changes would come a long way in terms of "Developer happiness"

To add my feedback, we've been using go in our monorepo and I think the confusion fundamentally boils down the decision that workspaces are an override for local development, but according to guidelines shouldn't be production, and therefore go mod and go work are orthogonal.

We have a setup like:

./services/...
./packages/go/...

We used to have a go.mod at the top level, but that made every package share the same set of dependencies. Creating a workspace seemed like a nice approach (like yarn workspaces in JS for monorepos), and then we'd set up each package in services and packages/go with their own go.mod file. The result is that we have to micromanage it with replace directives now, and those can only be added by hand or through the CLI.

We also tried deleting the workspace and running go mod tidy again, and it doesn't understand imports from other parts of the same repo, yet the code builds successfully.

From my perspective, I feel like the tooling is unintuitive and is probably the most 'magical' part of the language, considering how methodical and obvious the rest of the language is.

I've encountered a similar problem. the current solution is write a shell to go mod replace first and then go mod tidy and removing go mod replace. Hope GO Support go mod tidy Ignore go.work.

As someone who is starting with Go, and learning micro-services, I want to share my perspective, on this workspace thread.

Here's my user case that I was working on with workspace:

• created module micro-service-1.

• created module micro-service-2.

• created now a utils module, currently micro-services-1 uses it.

• created a gRPC module, it have generated code that allows ms-1 and ms-2 to talk to each other.

Now I want to build docker images for ms-1 and ms-2.

what would be my advantage for using workspace in this scenario?

• Faster download times for building images, for example, ms-2 don't use utils module so it don't need to copy it and download it's dependencies.
• Better mental model for external packages.
• from a newbe perspective this looked as the right way to do things.

now here's the problem that is emerging:

• having to manually tidy things
• now for tidy to work I have to do workaround with this: removing link style from module names, publishing as a package, using replace (learned about this now)

Too much trouble, probably I'll just stop using workspace and forget the possible advantages

My suggestion for a "go mod tidy --workspace", or "go work tidy".

It should recursively go to workspace modules and tidy them too.

For example, if inside ms-1 I run the supposed "go work tidy", it should see that it's using the utils module and recursively tidy utils too.

If this behavior is debatable put a flag for it e.g. "go work tidy --recursive".

@EduartePaiva I'm wondering why you didn't place all your packages in the same module? That is the happy path. And then no workspace is required.

To others on this issue: I would be like to speak to people about the requirements they have that push them towards multi-module repos with go.work files. I think if we could better understand those requirements we would be better equipped to see how to satisfy them.

Please let me know if you're interested in talking about your use case.

@matloob being honest I was following a microservice course and the teacher started using workspace to separate each microservice. At the time I didn't questioned why, just thought that it was the way to do things.

@matloob The main reason I was looking for a workspace solution is that a large module doesn't play well with creating docker images from both a caching perspective as well as a building perspective. Workspaces have some challenges here as well without jumping to a solution like Bazel to maintain build dependencies.

On top of not being able to independently version any libraries for external consumption.

Ultimately we did go with a common module and some complex docker contexts to make sure everything came together.

I am no longer at $company with a main repo. This plus things like github CI and the like really loves to make monorepos benefits start to dwindle a lot.

@matloob In my case I am rewriting several discord bots to go. However they have some shared code which I am putting in their own package & the services themselves.

You can check it out here

Now I got a dockerfile for each and I copy the mod file for the app, then the shared mod file and download them seperately etc.

Hi,
Working on https://github.com/DataDog/datadog-agent, we decided to split our repository in several modules to be able to import part of the agent code without importing the whole code base. It definitely worked but brought some challenge on the tooling to manage the multi module repository.
We recently introduced go.work and added it to the repository. It improves things, because we are now able to keep dependencies version synchronised between the modules and it simplifies running test in the different modules. But we have not been able yet to remove all the replace from the go.mod files because of we the things mentioned in this issue.

My suggestion for a "go mod tidy --workspace", or "go work tidy".

It should recursively go to workspace modules and tidy them too.

For example, if inside ms-1 I run the supposed "go work tidy", it should see that it's using the utils module and recursively tidy utils too.

If this behavior is debatable put a flag for it e.g. "go work tidy --recursive".

I did this using a Makefile to handle the tidy

@EduartePaiva Ah, got it. Yeah, in general multi-module repositories should only be reached only if a single repo repository isn't working for your use case.

@nemith Do you have modules in your workspace that aren't go get-able? And if so, what's the constraint that prevents them from being go get-able? My understanding is that that is when this issue occurs- when there are modules in your workspace that can't function independently.

@jurienhamaker Are the bots in separate modules to ease deployment similar to the problems @nemith mentioned?

@KevinFairise2 What were the downsides of importing the whole codebase? Was it to restrict the size of dependencies that were brought in with the imports? Also, are each of the modules independently go-gettable? If you're able to do a call I'd definitely be interested in talking about your use case.

@matloob The bots itself can function independently as long as they are compiled together with the shared module. The shared module is their only dependency.

The dockerfiles in the repository show basically that, I don't add service y & z to the docker container of x.

@matloob I am going to go off on memory here (maybe I will try to reproduce it if i get time later). But as I remember everything worked fine until I added a new module that wasn't yet committed to the repo.

Despite being listed in go.work no other modules could try to use it as it would try to contact the repo (github in my case) to get information on the imported repo instead of just looking locally first.

I guess that makes it not go getable but that was just because of order of operations and not desired end state.

If i remember right there were workarounds but all of it added too much cognitive load to the team who's job was to write software and not understand the rube goldberg building machine as much as I could.

@matloob one consideration is that Go is not necessarily the whole picture for any given project. We have a product with a front-end in Typescript, multiple backend services in Go, and an app in Swift. They all communicate using GRPC, and depend on a single common set of protobufs, from which we generate types, clients, and server stubs.

Keeping all the protos, services, apps, and frontends in one repo allows us to change protobufs, regenerate all the client and server stubs, and commit the consequences to the whole system all in one go. in particular, we can understand if anything is going to break before changes are merged to main. New versions of affected services and apps can also be automatically deployed based on changes to protos on which they depend.

All the go code being in one repo is a subset of that need, and similar to what others have said: common grpc (and other) code in lib/go/src and service code is services/service-one, services/service-two etc.

To make this work, we commit go.work to the repo (even though that's not its intended use). go.work and go.mod look like:

go.work:

use (
  ./lib/go/src
  ./services/service-one
  ./services/service-two
)

lib/go/src/go.mod:

module company.com/common

services/service-one/go.mod:

module company.com/service-one

Each service can then import "company.com/common/grpc" and everything resolves nicely. Building an application is simple too: go build company.com/service-one/app. We use go work sync to keep all the package dependencies consistent across services.

This all works very nicely. The only slight issue is that go mod tidy within a given service gives errors:

go: company.com/service-one/app imports
	company.com/common: cannot find module providing package company.com/common: unrecognized import path "company.com/common": https fetch: Get "https://company.com/common?go-get=1": dial tcp: lookup company.com: no such host

this is fine, because there's nothing to retrieve, but it would be ideal if it paid attention to the use directives of go.work and didn't bother to do the lookups.

As for why the services aren't all in one module together - it's because they aren't all one module. They are applications that happen to be written in go, amongst others that are not. They are each deployed separately. They have different environment variables, they have different kubernetes configs. We choose to organize our code by logical service, not by implementation language.

This is service one:

├── app
│   ├── main.go
├── env
│   ├── dev.env
│   ├── production.env
│   └── test.env
├── go.mod
├── go.sum
└── k8s
    ├── dev.yaml
    ├── production.yaml
    └── test.yaml

This is service three:

├── app
│   ├── index.ts
├── env
│   ├── dev.env
│   ├── production.env
│   └── test.env
├── package.json
└── k8s
    ├── dev.yaml
    ├── production.yaml
    └── test.yaml

The enforced regularity of this layout greatly simplifies devops in a heterogeneous repo, and reduces cognitive load on developers as the system and team grows.

@majelbstoat I'd like to understand this better. Is the goal of placing the go.mod files in each of the service directories (rather than the top level) to make the versions of the dependencies of the services independent of each other?

The idea is that each go.mod represents the minimal set of dependencies for each service/library. This helps optimize deployments, where we save the go build cache and results of go get for each service in a cache keyed by the go.sum of that service (and the common library go.sum).

.circleci.yaml:

restore_cache:
      keys:
          - go-mod-v3-{{ checksum "lib/go/src/go.sum" }}-{{ checksum "services/<< parameters.service-name >>/go.sum"}}

Practically speaking, that means if we make a change to the dependencies of service-one, we only have to expire the cache for service-one, not every go service. If they were all in one go.mod, we would have to redeploy every service, and blow every cache, for every dependency change to every service, even if the other services didn't depend on the new dependency. (Or else make our deployment decision-making process quite a bit more complex.)

We actually want common dependencies for each service to be at the same version. For example, it's important that the version of the google.golang.org/grpc package is the same for all services. go work sync does this for us - it ensures all go.mods that include google.golang.org/grpc include it at the same highest version. It gets run automatically pre-commit if any go.mod has changed.

Aesthetically too, each service imports packages from the common library. If they all were in the same go.mod, imports for a service from the common library would be like import "company.com/lib/go/src/common/grpc", which kinda ugly and a pain to change if we ever changed its relative location in the repo. It doesn't happen often, but it has happened. In the current setup, it requires one edit to one use line in go.work.

I think that I have a simpler reason for why this is incorrect behavior right now.
go.mod and go.sum are distributable artifacts. go.work is for local overriding of dependency modules for more streamlined development. Once development is done, I should not need to modify go.mod as I will have pushed my simultaneously developed dependency module as well as this one.

Expected behavior:
I have an as yet unpublished dependency which I have added to my workspace ../mymod. If I go get or go mod tidy it should not attempt to locate mymod upstream because it is defined in my workspace. It should continue to get other dependency modules not included in my workspace. On finishing development and publishing the dependency, I remove the workspace and go get to update the dependencies in go.mod and go.sum

Observed behavior:
go mod tidy and go get retrieve dependency modules but fail on trying to load my unpublished dependency. The newly retrieved modules are not added to go.mod and go.sum leaving the external dependencies unresolved.

Workaround:
add replace myrepo.myid.mymod => ../mymod which we used to do before workspaces. This is unsatisfactory because once development is done, you need to go back into go.mod and remove this override.