Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: issues with shared $GOPATH/pkg/mod cache and autogenerated modules #32235

Closed
sirkon opened this issue May 24, 2019 · 3 comments
Closed

cmd/go: issues with shared $GOPATH/pkg/mod cache and autogenerated modules #32235

sirkon opened this issue May 24, 2019 · 3 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.13

Comments

@sirkon
Copy link

sirkon commented May 24, 2019 •
edited

Hi!

At my job we have a single repository with proto-files grouped by directories. Something like gitlab.example.com/common/schema/<service path>/<file.proto>

We were generating this into $GOPATH/src/gen/<service path>/<file.go> and import this as gen/<service path>

Now we did a server which acts like a good go modules proxy except the case when it is requested for gitlab.example.com/common/schema/<service path>/v34. It generates go code from gitlab.example.com/common/schema/<service path>/*.proto files and return them as regular go module. Sounds nice and it works great in most cases except one thing: most our projects use code generated with protoc-gen-go, but some are playing with code generated by protoc-gen-gofast.

Our decision was to have two URL serving goproxies: https://goproxy.example.com/go and https://goproxy.example.com/gofast where the first one requests for 3rd party goproxies and returns code generated with protoc-gen-go and the other one works exactly the same except it would return code generated with protoc-gen-gofast. We thought we would simply use different GOPROXY for different projects and everything will be great.

And this doesn't work that easy unfortunately: we have shared $GOPATH/pkg/mod – cache directory for downloaded modules. And we will have exactly the same autogenerated module of certain version for project A if we have already requested this version of the module for project B despite having different sources.

Sure, we can manually clean cache dir but this is … ugly. We would probably have different cache directories with different GOPATH, but we would prefer to keep it the same.

So I have a proposal: a new environment variable called GOCACHE or GOMODCACHE or whatever to control modules cache. This would ease the development if editor/IDE supports project profiles.
@thepudds
Copy link
Contributor

I think I might have misunderstood the problem, or at least I didn't follow this paragraph:

And this doesn't work that easy unfortunately: we have shared $GOPATH/pkg/mod – cache directory for downloaded modules. And we will have exactly the same autogenerated module of certain version for project A if we have already requested this version of the module for project B despite having different sources.

Could you expand slightly on the actual problem you encountered?

We would probably have different cache directories with different GOPATH, but we would prefer to keep it the same.

Can you expand why different GOPATH wouldn't work for you?

@thepudds thepudds changed the title Issues with shared $GOPATH/pkg/mod cache and autogenerated modules cmd/go: issues with shared $GOPATH/pkg/mod cache and autogenerated modules May 26, 2019
@sirkon
Copy link
Author

sirkon commented May 27, 2019 •
edited

Could you expand slightly on the actual problem you encountered?

Our situation: we have service called marker whose gRPC declarations are defined in gitlab.example.com/common/schema/marker and we have web server with autogenerated handlers providing REST interface to some gRPC handlers which is using these declarations to access marker.

But the difference is marker works with code generated with protoc-gen-go, while the web server working with code generated by protoc-gen-gofast because of some additional functionality we needed (finer JSON marshaling control mainly).

Both are using helpers what import gitlab.example.com/common/schema/marker and thus work both for protoc-gen-go and protoc-gen-gofast generated code.

Can you expand why different GOPATH wouldn't work for you?

It would actually, we would just prefer to keep it the same for things like go install.

@julieqiu julieqiu added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label May 28, 2019
@julieqiu julieqiu added this to the Go1.13 milestone May 28, 2019
@bcmills
Copy link
Contributor

bcmills commented May 29, 2019

We thought we would simply use different GOPROXY for different projects and everything will be great.

Sorry, but that's not how GOPROXY works. It is a proxy, not a source of alternative truths.

As you've observed, every proxy must serve the same contents for a given module path and version; that is fundamental to the design of modules (and to the reproducibility of module-enabled builds), and enforced by checksums in go.sum and the database enabled in #25530.

As an alternative, you might be able to serve the two variants at different module paths and let users apply their own replace directives to substitute one for the other.

@bcmills bcmills closed this as completed May 29, 2019
@golang golang locked and limited conversation to collaborators May 28, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.13
Development

No branches or pull requests
5 participants
@sirkon @bcmills @julieqiu @gopherbot @thepudds