Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: setting up a multi-module single repo is difficult #27056

Closed
hyangah opened this issue Aug 17, 2018 · 11 comments
Closed

cmd/go: setting up a multi-module single repo is difficult #27056

hyangah opened this issue Aug 17, 2018 · 11 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@hyangah
Copy link
Contributor

hyangah commented Aug 17, 2018

Let's say we have a repository containing three packages and
we decided to organize them in separate modules for some reasons.
The purple package imports blue and red packages.

> cd $WORKSPACE/example.com/purple
> tree
.
├── blue
│   ├── blue.go
│   └── go.mod
├── go.mod
├── purple.go
└── red
    ├── go.mod
    └── red.go

We created go.mod files in each package directory using 'go mod init example.com/purple/red', etc.
Now we try to build the purple package.

$ go build all
go: import "example.com/purple" ->
	import "example.com/purple/blue": cannot find module providing package example.com/purple/blue
go: import "example.com/purple" ->
	import "example.com/purple/red": cannot find module providing package example.com/purple/red

What's happening underneath is, the go build command queries https://example.com/purple/red?go-get=1, ... to get the meta tags. That means, until I check in the packages and go.mod files to the remote repository and make them accessible, I can't build/test.

What's the easiest way to start a multi-module repo with inter-dependency?

@hyangah hyangah changed the title cmd/build: setting up a multi-module single repo is difficult cmd/go: setting up a multi-module single repo is difficult Aug 17, 2018
@rsc
Copy link
Contributor

rsc commented Aug 18, 2018

A go.mod is like its own little GOPATH. There is no implicit reference to other nearby modules. In particular being in one repo does not mean that they all move in lock step and always refer to the code from the same commit. That's not what users will get either. If there is an inter-dependency, you need to add it, with something like

cd $WORKSPACE/example.com/purple
go mod edit -replace example.com/purple/red=./red
go mod edit -replace example.com/purple/blue=./blue

@hyangah
Copy link
Contributor Author

hyangah commented Aug 29, 2018

This issue is closely related to #26241, #25053 and other issues that involving development of multiple modules concurrently and access to local modules.

In the above example, one caveat is that the 'replace' rule doesn't take effect until there exist the 'require' entry for the replaced module. That's why I ended up using 'go mod edit -require' which is not meant for use by human. (#27060) So, in order to add the local inter-dependency, users need to either

  1. commit the updated red, blue modules upstream so they can be go-gettable. Then transform purple, or
  2. manually edit go.mod to include red and blue entries both in 'require' and 'replace'.

At the beginning of converting an existing repo to a multi-module repo, currently, users will need hand-edit go.mod file (go mod edit -require and go mod edit -replace) and careful planning of commit ordering.

@andybons andybons added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Sep 4, 2018
@andybons andybons added this to the Unplanned milestone Sep 4, 2018
@andybons
Copy link
Member

andybons commented Sep 4, 2018

@bcmills

@bcmills
Copy link
Contributor

bcmills commented Sep 4, 2018

This issue is closely related to #26241, #25053 and other issues that involving development of multiple modules concurrently and access to local modules.

@rogpeppe's gohack tool has some nice prototyping for hacking on multiple modules. I wonder whether that supports multi-module repos, and if so what we can learn from it.

@myitcv
Copy link
Member

myitcv commented Sep 7, 2018

@hyangah I submitted an experience report for a similar setup.

From my perspective, the only "problem" is the requirement on existence of the submodule (and I don't think we can solve that problem). For me, this was step 1. But this was only a "problem" in so far as my CI build broke. Beyond that commit, all was fine.

At the beginning of converting an existing repo to a multi-module repo, currently, users will need hand-edit go.mod file (go mod edit -require and go mod edit -replace) and careful planning of commit ordering.

This is effectively the crux of my open questions too. I'm less worried about the "hand-edit" part, because I think tooling can easily fix that. It's the commit dance that follows that raises the biggest question. Again, tooling could help, but because of the mix of VCS systems that people may be using, it's not immediately obvious. With Gerrit things are certainly much easier...

@zellyn
Copy link

zellyn commented Nov 8, 2018

As an experience report, our Go code at Square is organized in a similar fashion. We are still using godep because we were waiting for the Packaging Wars to settle out.

We have O(dozens) of apps at $REPO/$APPNAME or $REPO/$TEAM/$APPNAME, as well as components like $REPO/service/logging, $REPO/service/feeds, $REPO/service/rpc, etc.

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo), but we're considering giving each app a separate go.mod file: it would allow each app to choose what version of third-party libraries it uses.

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that all go.mod files include things like replace directives for our protobuf fork. I was intending to have that tool enter replace $REPO/service/foo => ../service/foo directives for each module imported, or perhaps a global replace $REPO => .. to catch all of them. It's awkward, but not unworkable: those replace directives would be out-of-place in the checked-in code if this were an open-source repo, but since it's our monorepo, everything would probably work okay with them checked in.

Anyway, just wanted to log a report. I expect within Google, Go code is still built using blaze, with custom tooling, so I don't expect monorepo experience help from inside :-)

@myitcv
Copy link
Member

myitcv commented Nov 8, 2018

@zellyn just to check on one point:

It doesn't seem as if hosting multiple modules out of a single repo is clean (version tags are per-repo)

Do you mean your version tags are per-repo? Because in general multi-module versions work as a result of the submodule prefix on a tag. For example the repo that results from the submodules (multi-module repo) guide has the following two tags (i.e. versions):

b/v0.1.1
a/v1.0.0

We're also considering some kind of linter/auto-go.mod-writing tool ...

This would presumably be a wrapper around go mod edit -replace... and friends?

@zellyn
Copy link

zellyn commented Nov 8, 2018

@myitcv

Because in general multi-module versions work as a result of the submodule prefix on a tag.

Ah, I somehow missed that you could namespace version labels like that. Simple and clear. Thanks!

This would presumably be a wrapper around go mod edit -replace... and friends?

I imagine so, probably. The more important reason for linting is that we need to catch an app accidentally not using our protobuf fork. It could result in not redacting sensitive fields in logs, for example.

@bcmills
Copy link
Contributor

bcmills commented Nov 8, 2018

We're also considering some kind of linter/auto-go.mod-writing tool to ensure that all go.mod files include things like replace directives for our protobuf fork.

That seems like a problem you could resolve with a GOPROXY HTTP server, at least: it could act as a straight proxy for most packages (or serve 404s after #26334 is fixed), but serve 403s for any module you want to exclude globally.

@zellyn
Copy link

zellyn commented Nov 9, 2018

Interesting idea. I'll have to see if we can get our Artifactory instance to do that, after we get the basic functionality working).

@bcmills
Copy link
Contributor

bcmills commented Jan 18, 2019

We've addressed a number of the underlying issues, although there are certainly still some difficulties in setting up and testing multi-module repos.

Let's track the remaining issues individually rather than under this umbrella issue.

I know that at least the following are related:

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

7 participants