On macOS and Linux, both Go 1.24 and 1.23 didn't work as expected:

% go version
go version go1.24.0 darwin/arm64
% go run github.com/hajimehoshi/overlaytest 
Original
% cat overlay.json
{"Replace":{"/Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/var/folders/cj/73zbb35j0qx5t4b6rnqq0__h0000gn/T/bar.go1091616997"}}
% go run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

% go1.23.6 run github.com/hajimehoshi/overlaytest 
Original
% cat overlay.json
{"Replace":{"/Users/hajimehoshi/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/var/folders/cj/73zbb35j0qx5t4b6rnqq0__h0000gn/T/bar.go4028295237"}}
% go1.23.6 run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

$ go version
go version go1.24.0 linux/arm64
$ go run github.com/hajimehoshi/overlaytest
Original
$ cat overlay.json
{"Replace":{"/home/parallels/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/tmp/bar.go2784474435"}}
$ go run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

$ go1.23.6 github.com/hajimehoshi/overlaytest
Original
$ cat overlay.json
{"Replace":{"/home/parallels/go/pkg/mod/github.com/hajimehoshi/overlaytest@v0.0.0-20250216151013-e210b75fba9a/internal/foo/bar.go":"/tmp/bar.go1458768758"}}
$ go1.23.6 run -overlay=overlay.json github.com/hajimehoshi/overlaytest
Original

EDIT: My current guess is that replacing files in module cache might cause some troubles.

CC @matloob per #71075 (comment)

ping @matloob ?

(We had a US Holiday yesterday)

Yes, we don't respect overlays in the modcache. We should explicitly reject those cases.

I'll send a CL to explicitly reject them. But that CL wouldn't meet the bar to be backported, so it would show up in Go 1.25.

Thank you for replying!

Yes, we don't respect overlays in the modcache. We should explicitly reject those cases.

1. Is there a way to use -overlay for external modules?
2. Is it guaranteed that -overlay always works for the standard libraries?

It was suggested in #69887 (comment) that -overlay be used to modify code being built, e.g. for instrumenting code, in place of -toolexec. Blocking -overlay in dependencies seems like it'd greatly limit that approach (and potentially make it hard/impossible to use this method on projects split into multiple modules).

That and, -overlay powers gopls, so typing in the module cache (accidentally!) will probably lead to some strange effects when the open file overlay is used, or crashes if the go command bails out, right?

Just FYI, my current -overlay usages are

1. to separate code under NDA for private repositories (e.g. Ebitengine's implementation for consoles)
2. to modify the standard library for consoles (https://ebitengine.org/en/blog/native_compiling_for_nintendo_switch.html)

@hajimehoshi To answer the easier question first, yes -overlay works for the standard library. That shouldn't change.

On the other hand the module cache is intended to be read-only (we write the files read-only on disk) so the Go command assumes that it isn't modified, even by an overlay.

@jakebailey I think we would still be able to find a solution even if we don't use overlays. For instance, we might be able to use replace directives. We would plan to have gopls filter out the files provided to the overlay to avoid a crash. See #71075 (comment)

@hajimehoshi I think replace directives are the best option for replacing code that's in the module cache. Would they work for your use cases? For the ebitengine NDA use case, could you replace the module containing the NDA'd code when doing the build in an environment where the implementation is available? You could perhaps split the specific interfaces into their own module and replace only that module to avoid having to replace a big module.

If it becomes necessary we can reverse course on this: for example, we could disable the module index when an overlay modifies the module cache. That should fix the immediate problems. But it may create other ones if we start breaking the assumption that the module index is immutable, so I think we should avoid that option if we can.

@matloob Thanks! I'm relieved that -overlay for the standard library is fine!

I think replace directives are the best option for replacing code that's in the module cache. Would they work for your use cases? For the ebitengine NDA use case, could you replace the module containing the NDA'd code when doing the build in an environment where the implementation is available?

It is possible, but this means that a user has to modify their go.mod for their applications. Perhaps, I can develop a tool to do it automatically as a wrapper of a Go compiler, so this might work. It'd be like gomobile/gobind, which creates go.mod in a temporary directory, creates necessary sources there automatically. I'll try, thanks!

You could perhaps split the specific interfaces into their own module and replace only that module to avoid having to replace a big module.

This might work, but I don't want to expose the internal details as an exposed interface. Another concern is that the interface content can be changed anytime.

If it becomes necessary we can reverse course on this: for example, we could disable the module index when an overlay modifies the module cache. That should fix the immediate problems. But it may create other ones if we start breaking the assumption that the module index is immutable, so I think we should avoid that option if we can.

That makes sense.

Alright, I am convinced that -overlay for Go 1.23 on Windows worked for external modules but this was just a coincidence. Thank you for your insights!

Change https://go.dev/cl/650475 mentions this issue: cmd/go: explicitly reject overlays affecting GOMODCACHE

I've created my own helper library to overwrite files in external libraries: https://github.com/hajimehoshi/uwagaki