Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

proposal: cmd/go: allow replace directives in go.mod to work for standard libraries #35283

Closed
zachwalton opened this issue Oct 31, 2019 · 16 comments

Comments

@zachwalton
Copy link

This may be completely incompatible with Go's architecture, but I'm putting it out there as a request for comments :)

Basically, make this work:

09:56 $ cat go.mod
module github.com/my/repo

go 1.13

replace (
	crypto/tls => ./third_party/crypto/tls
)

This would allow for forking standard libraries without requiring users to use a forked version of Go. In my case, I've implemented #22836 and would like to use it in my project without maintaining a full Go fork.

What version of Go are you using (go version)?

09:56 $ go version
go version go1.13 darwin/amd64
@gopherbot gopherbot added this to the Proposal milestone Oct 31, 2019
@networkimprov
Copy link

Or you can patch your stdlib, which a number of projects do.

I'd liike to see Go support for patching stdlib & runtime, e.g. an env variable giving a patchfile or directory, and a way to amend the version string to "1.13.3_myproject-1.1"

@rsc
Copy link
Contributor

rsc commented Nov 6, 2019

This is in general not possible. The standard library is not a collection of individual modules. It is more like a single module. That is, the packages in the standard library are very tightly coupled. You can't mix and match some from one version of Go and some from another, nor some hacked up and some not.

@rsc rsc changed the title Proposal: Allow replace directives in go.mod to work for standard libraries proposal: cmd/go: allow replace directives in go.mod to work for standard libraries Nov 6, 2019
@networkimprov
Copy link

@rsc, what do you think about Go support for patching stdlib & runtime, e.g.

  1. a build-time env variable giving a patchfile or directory, and
  2. a way to append to the runtime.Version() result: "1.13.3_myproject-1.1"

@jayconrod
Copy link
Contributor

To expand on @rsc's comment, replace directives in go.mod control modules, not packages. Modules are the unit of versioning and distribution, so it doesn't really make sense for them to apply to individual packages (and as with std, doing so would break many modules).

@networkimprov Why not just patch GOROOT directly and build a toolchain from there? That seems far simpler.

@networkimprov
Copy link

@jayconrod $GOROOT contents may be discarded on upgrade. And any patches are invisible at build-time. You wouldn't want everything you build to carry patches necessary for a certain app.

I currently patch my $GOROOT directly, but runtime.Version() does not reflect this fact. I looked for a way to amend the .Version() result, but didn't come up with something straightforward. I forget why offhand...

@jayconrod
Copy link
Contributor

@networkimprov

$GOROOT contents may be discarded on upgrade.

There are a lot of ways to handle this. For example, as commits on a branch that gets rebased after upgrading. Or as a list of files applied by a bash script.

I looked for a way to amend the .Version() result, but didn't come up with something straightforward.

I think you can write to $GOROOT/VERSION before running make.bash. See findgoversion in src/cmd/dist/build.go for details.

In general though, anyone who's patching the standard library or toolchain in their project has a fairly specialized use case, and I wouldn't expect one solution to work for everyone.

This is a bit off-topic from the original issue in any case (replace on standard packages), so let's table this part of the discussion.

@zachwalton
Copy link
Author

Thanks for the thoughtful replies. It sounds like the premise of my proposal was off-base, so more generally:

I think it'd be helpful to have a portable way to patch standard packages imported by a Go project, "portable" in this sense meaning being able to use a module that patches standard packages with nothing more than go get github.com/module/that/patches/stdlib.

Is there a potential place for some mechanism along those lines in Go?

@networkimprov
Copy link

@jayconrod since the original suggestion isn't achievable, I'd like to hear from @rsc whether a patch-support proposal is worth investing time in. It would solve the stated problem.

Patching runtime or stdlib should not require a git-clone of the Go project, nor a toolchain build from source.

A number of Windows users currently duplicate stdlib code in order to patch it; see #34681.

@jayconrod
Copy link
Contributor

I wouldn't support this proposal. It would result in modules running slightly different, incompatible dialects of Go. That would prevent the ecosystem from scaling and would cause problems for large projects.

It's also not clear how this would be implemented. Would go build rebuild the toolchain and re-exec itself when the patches change?

I'm not familiar with the details of file flags in #34681. My (somewhat uninformed) opinion is that if a standard library package isn't working as desired, it should either be fixed (assuming it won't break other users) or forked.

@rsc
Copy link
Contributor

rsc commented Nov 13, 2019

If you want to patch the standard library, I would say go ahead and do that.
Adding explicit support to encourage that behavior seems like a step in the wrong direction.
In general figuring out how to let people experiment with changes is an open question,
but we know that one way to do it wrong is to make it too easy to let production code depend
on things like this that break upgrades and so on.

This proposal is a likely decline: it's too invasive for too little benefit.

@networkimprov
Copy link

networkimprov commented Nov 13, 2019

One reason many projects need patches is that each Go release stops getting regression fixes after 6 months (see #34622).

What would you think of a proposal re adjusting the runtime.Version() result to reflect a non-released (e.g. patched) instance?

@mpx
Copy link
Contributor

mpx commented Nov 14, 2019

You can set the Version by creating a $GOROOT/VERSION file and building the toolchain (see src/cmd/dist/build.go:findgoversion for details).

I've found it's relatively easy to maintain a custom branch (eg, go1.13.4mpx) and tags when I want to use newer features but need (otherwise outstanding) fixes to the stable tree. This works, provided you don't want to distribute source to people who don't use your branch.

@networkimprov
Copy link

Thanks, but patching runtime or stdlib should not require a git-clone of the Go project, nor a toolchain build from source (which I mentioned above :-)

I currently distribute a stdlib patch in my releases, which has to work after
$ cp x.patch .../go && (cd .../go && git apply x.patch).

@rsc
Copy link
Contributor

rsc commented Nov 20, 2019

I forgot to add the final comment label last week. This is still a likely decline.
Leaving open for one more week for final comments.

@bcmills
Copy link
Contributor

bcmills commented Nov 25, 2019

If people want to replace specific parts of the standard library, I think #30241 or something similar would provide a more consistent way to do that.

If we decide that a given standard-library package should be independently upgradeable, we can make that package forward to a corresponding package in some other, actual module (such as golang.org/x/crypto). Then, we can allow the module dependencies of the standard library to be upgraded and/or replaced, rather than replacing arbitrary chunks within the standard library.

@rsc rsc added this to Final Comment in Proposals (old) Nov 27, 2019
@rsc
Copy link
Contributor

rsc commented Nov 27, 2019

Consensus did not change, so declining.

@rsc rsc closed this as completed Nov 27, 2019
@golang golang locked and limited conversation to collaborators Nov 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants