Thanks for opening the issue :) For those wanting a link to the CL above, it's https://golang.org/cl/151999.

The current CL currently extends Overlay map[string][]byte. The one problem with that approach is that the user must know all the files to add beforehand.

For example, I'd like to be able to add some Go files to directories that have a special non-Go file. So if I do Load("./..."), I can't possibly know which files to add beforehand. I could possibly do something like go list -e -f {{.Dir}} on my own, filter the directories containing that special file, then add the custom Go files to the overlay map.

But of course that would hard-code go list and require quite a bit of code. Could it be possible to have some form of overlay that works like a callback, so that I could add Go files on a per-package or per-directory basis? Ideally it would support introducing new packages too.

I know that's perhaps asking for too much, but I want to clarify that the current approach won't quite cut it for some use cases :)

I have thought about this use case.

The trouble with the callback based overlays is it does not work with an external driver. If you look at the work that Marwan did on a caching version of the go list driver, there is no way to have that separate process cope with a function based overlay in a way that is even slightly efficient, which is why we chose a pure data model that can be handed across a process boundary.

For the moment, you can call Load twice, once in Files mode, and then again in Syntax mode with the constructed overlay.
It's not quite as efficient as it could be, but the functionality is equivalent, and once we have some real use cases we can think about API's that allow the operations people really need to do to become more efficient. Also the Files mode will be very very fast once the new caching code goes in to the go list operation, to the point where it's probably good enough anyway.

Thanks for the reply - that indeed does work OK when it comes to adding or changing Go files within existing Go packages. But as far as I know, that can't be used to construct new Go packages easily.

For example, suppose my special file per package is called foo.bar and introduces a foo.go file. With the go list driver, if we load ./... and a directory consists of the files foo.bar main.go, your approach will work just fine. But if a directory only contains foo.bar, I believe that LoadFiles will skip that directory, just like go list ./... would.

I also just realised that it's not possible to list non-package directories with go list -e -f {{.Dir}} ./..., either. I wish that either go list or its go/packages driver had a "find directories" mode, where patterns like ./... would be resolved to lists of "potential packages" before they're discarded for not having any Go files.

Also, I do realise that my talking of "Go packages" as "directories" is quite particular to go list, and not necessarily other build systems. So I'd understand if all this was out of scope for the current plans for go/packages. Hopefully I managed to illustrate why I still can't accomplish what I need, though.

@mvdan I'm not sure that go list will be able list non-package directories, so I don't know if this sort of thing will be supported first-class. So solving your problem might require traversing all the relevant directories to find the .bar files and constructing an overlay based on that. I know that's not satisfying, but I don't see a way around it. To make things worse: as you mentioned, it's not clear how we'd make something like this work with alternate build systems

Thanks - I agree this is an awkward use case :)

Manually traversing directories can be a tricky solution though, as it essentially means I need to reimplement go list patterns myself. For the simple ./... I could just filepath.Walk, but what about trickier ones like .../foo or .../foo/...?

It's completely fair if this is out of the scope of go/packages, and I do understand I'd be tying my implementation to go list quite a lot - and that's fine. But I'd still need a way for go list to give me directories matching patterns, not just Go packages. This is what I thought go list -e -f {{.Dir}} ../foo would do, which is why I gave that example in my first comment - but it too skips directories with no Go files.

So I should really be filing an issue against go list :) I'd like your thoughts before raising the formal proposal for the feature, though.

I suspect such an issue would be denied on the assumption that it is the go tools job to deal with go code, so returning information about directories that have no go code is out of scope.
Is there really a use case for this where just saying there must be a doc.go with a valid package statement is not an acceptable solution?

I suspect such an issue would be declined too, which is part of the reason why I'm approaching it from the perspective of overlays creating new packages :)

We are indeed forcing users to add doc.go files to each of those directories. We end up with Go-like packages on disk with no actual static Go files. If we force users to add doc.go, that's extra manual work and exposes impelmentation details. If we implement our own go list pattern matching, that will likely lead to lots of added code. I don't see a simple solution.

Perhaps the solution would be a Go library similar to the older https://github.com/kisielk/gotool, but where you'd be able to configure things such as whether a directory should be a package or not. That would still require me to copy some code from go list though, so I'm still hopeful that there's a way to support this in a simple way.

Creating the doc.go is easy enough to write a small tool for, it does not need to be a manual step, just build a doc.go for any directory that has a .bar file and no .go files before doing any list queries.
Presumably the contents of the doc.go file could easily be guessed from the .bar file.
I guess I still don't really understand your use case properly either, what you are describing is something that would never work with anything else in the go eco system anyway (you can't use any of the normal go tooling on a directory with no go files, including go generate or go get)

Correct. We're hacking our way around these problems with overlays, so I'm moderately happy with the situation. If I come up with a specific proposal for go/packages to improve our use case, I'll file a separate issue.

Several other programming languages solve this by having a specific file that can list the files that belong to a package. We already have go.mod, so in the same vein, we could use a go.pkg file that lists patterns of files to ignore and of files to include as go file.

Closing this issue because new packages overlays are supported and support for overlays will be on a solid foundation with Go 1.16.