Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

embed: document go.mod exclusion #45197

Open
codefromthecrypt opened this issue Mar 24, 2021 · 16 comments · May be fixed by #61976
Open

embed: document go.mod exclusion #45197

codefromthecrypt opened this issue Mar 24, 2021 · 16 comments · May be fixed by #61976
Labels
Documentation help wanted NeedsFix The path to resolution is known, but the work has not been done.
Milestone

Comments

@codefromthecrypt
Copy link

We have a use case for serving example go projects via go:embed. This is particularly important as alternatives to go:embed have less maintenance now that they are obviated. The problems is that the file go.mod is special cased and prevents serving a directory that contains it.

@opennota did research here and found this behavior to be intentional, notable "stopping at module boundaries" #41191 (comment)

When serving static assets, module boundaries may not be important and in fact be a surprising limitation, as surprising as files outside the directory or via symbolic links might be. Right now, the latter constraints like symbolic links are documented, but go.mod, and why it doesn't work is not entirely obvious.

Ideally, there could be a way to allow go projects to serve go projects as static assets via go:embed. Until then, it would be handy to change the documentation to note this surprising constraint explicitly, as it would save folks time and head scratching.

Ex a line like this "Matches for a directory containing "go.mod" is invalid and fail to build".

What version of Go are you using (go version)?

1.16.2

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

amd64 darwin

What did you do?

//go:embed /path/to/goproject/*
var exfs embed.FS

What did you expect to see?

I expected to have go build not fail

What did you see instead?

An error like "pattern X: cannot embed directory Y in different module"

@Merovius
Copy link
Contributor

Note that the primary distribution mechanism of modules are zip-files, which will only include the files that are considered part of the module. As I understand it, the zip you download from the module proxy (as opposed to using the repo) won't contain the nested module. So, module boundaries are very important - you can't embed a file that isn't there.

A workaround for your case is to name the file differently - e.g. go.mod_ - and rename it after the fact by wrapping the embed.FS.

@jayconrod jayconrod changed the title Support or document go.mod exclusion for go:embed embed: document go.mod exclusion Mar 24, 2021
@jayconrod jayconrod added this to the Go1.17 milestone Mar 24, 2021
@earthboundkid
Copy link
Contributor

Name rewriting seems to come up a fair amount as a workaround for edge case problems with embedding. Someone should write a module to wrap an io.FS in a name rewriter. I volunteer to click the star button on Github and then link to it in comments when someone brings up an issue solved by rewriting, when someone else creates it.

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 25, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
@cagedmantis cagedmantis added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Mar 25, 2021
@cagedmantis
Copy link
Contributor

/cc @rsc

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 26, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Mar 26, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 3, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 6, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 6, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
@codefromthecrypt
Copy link
Author

one problem with the rename go.mod to go.mod_ approach is that running go mod tidy in the root, ends up importing whatever is in go.mod_. I'm not sure how to work around this, but it is annoying.

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

@Merovius
Copy link
Contributor

Merovius commented Apr 6, 2021

It would be nice to have a clean way to allow go:embed to opt-out of interpreting go.mod...

That can't work. It would mean your module can't be build from the zip-file uploaded to the mirror. Third-party embedding solutions don't have this problem only because they generate code which is compiled in - so they effectively create a copy (but in Go syntax) of the files from the nested module in the current module, so it's not a problem if the nested module isn't distributed.

This really can't be solved on the //go:embed side, as far as I can tell. It might be possible to add a way to distribute subdirectories with a go.mod without interpreting them as a module - I don't know. But as long as the files are not part of the module, //go:embed can't use them, full stop.

@codefromthecrypt
Copy link
Author

great explanation, @Merovius. I didn't quite get it last time, but I do now, both the what and the why. awesome!

codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 7, 2021
This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Signed-off-by: Adrian Cole <adrian@tetrate.io>
codefromthecrypt added a commit to tetratelabs/func-e that referenced this issue Apr 7, 2021
…129)

This reduces build complexity by eliminating a generation step with go:embed. This implicitly reduces tech debt even more because not only were we using a stalled project, statik, but also a fork of it.

go:embed is not perfect, as it disallows the file name go.mod. The workaround is to rename it to go.mod_ per golang/go#45197. While this is imperfect an if-statement is a far better punch than a dependency on a fork of a stalled project which also requires a codegen step.

Co-authored-by: Takeshi Yoneda <takeshi@tetrate.io>
Signed-off-by: Adrian Cole <adrian@tetrate.io>
@ianlancetaylor ianlancetaylor added help wanted NeedsFix The path to resolution is known, but the work has not been done. labels Apr 19, 2021
@ianlancetaylor ianlancetaylor modified the milestones: Go1.17, Backlog Apr 19, 2021
@gopherbot gopherbot removed the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 19, 2021
@crhntr
Copy link

crhntr commented Aug 2, 2021

I ran into this and decided to use go:generate to tar the embedded module and then include the tape-archived file. It worked. https://github.com/crhntr/disaster-recovery-acceptance-tests/blob/embed-fixtures/fixtures/fixtures.go#L22-L28

My solution looked something like this:

repo/
  cmd/
    test-app/
      go.mod
      go.sum
      main.go
  fixtures/
  fixtures.go
  go.mod
  go.sum

In fixtures.go I have something like:

var (
	//go:embed fixtures
	fixtures embed.FS
)

//go:generate tar cf fixtures/test-app.tar ./cmd/test-app

I then commit the fixtures directory.

If there was a zip or txtar FS implementation I would use that instead but tar works. For my use case, I have a helper function to write the tar to the filesystem in a tmp dir for each test that needs the test app.

@earthboundkid
Copy link
Contributor

For the record, there are zip and txtar fs.FS implementations.

@wagslane
Copy link

wagslane commented Jun 1, 2023

My use-case is that I'm building a server (in Go ofc) that serves Go projects as downloadable zip files. I have a workspace structure like this:

go.work
go.mod
main.go
projects/
  project-1/
    go.mod
    main.go

I want the top-level main.go to be able to embed nested project directories so I can zip them up at runtime and serve them over the wire. The reason it's convenient to keep the nested file named go.mod is because the go.work file at the top of the repo makes it easy to make updates to those nested projects without breaking tooling.

Are there any simple solutions for what I'm trying to do here? It would be so nice if there was a directive that allowed nested modules to be embedded (esp because they aren't a part of the package embedding them, they are essentially flat files)

Edit: My workaround currently is to have a build step where I zip up the projects dir into a projects.zip and commit that tho source control. Then I embed the zip file as a slice of bytes and implement a virtual file system on top of that... It kinda sucks but it works. I'd love to see this changed!

qiulaidongfeng added a commit to qiulaidongfeng/go that referenced this issue Aug 12, 2023
Fixes: golang#45197
Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
@qiulaidongfeng qiulaidongfeng linked a pull request Aug 12, 2023 that will close this issue
@gopherbot
Copy link

Change https://go.dev/cl/518858 mentions this issue: embed: document go.mod exclusion

@oktalz
Copy link

oktalz commented Aug 12, 2023

it really is interesting, you can embed any kind of code (python, rust, c#, ...)
but you can't embed go code, and to be precise, you can embed go code, but not if you use go.mod

my solution (and it is a bad solution) is to have a makefile that

  • rename go.mod to go.mod.embed (then internally while serving i rename it back) - then you can embed it
  • do a go build with embedded go files
  • rename go.mod.embed back to go.mod
    purely because only go code projects are making problems (to repeat again, all other languages you can embed)

ziping it seems awkward although a bit easier to to.

newly added gonew tool is also not a good solution (but it is a good option) because does not support air-gapped environment

@ianlancetaylor
Copy link
Contributor

While I'm sure the documentation could be improved, this is already documented. The docs already say "Patterns must not match files outside the package's module". If a subdirectory has a go.mod file, then it is by definition in a different module.

qiulaidongfeng added a commit to qiulaidongfeng/go that referenced this issue Dec 31, 2023
Fixes: golang#45197
Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
qiulaidongfeng added a commit to qiulaidongfeng/go that referenced this issue Jan 6, 2024
Fixes: golang#45197
Change-Id: I61dff71ba02bfafd054408d73b8a15db745d0198
metafeather added a commit to metafeather/tools that referenced this issue Jan 30, 2024
@francois76
Copy link

In go 1.18, the prefix "all:" has been added for the embed directive, to include dotFiles (for instance). However, go.mod directories are still not included. Maybe the "all:" prefix could also include these files too.

@seankhliao
Copy link
Member

as explained above, modules inherently cannot contain other modules.

@fgm
Copy link

fgm commented Mar 23, 2024

One thing which makes it weird is that AIUI the contents of testdata directories are not compiled so they should not be considered modules, but they still are.

/
/go.mod
/demo.go
/testdata
/testdata/go.mod
/testdata/other.go

In this layout, testdata is skipped by ./... patterns like go list ./... but if demo.go tries to embed testdata/ it is still considered as another module although it is not handled as code.

@Merovius
Copy link
Contributor

Merovius commented Mar 23, 2024

"Not being compiled" is not the same as "not being part of the module". They are still used in tests and tests can be run on downloaded modules, so testdata needs to be included in the module zip.

And yes, you can put additional files, outside of the module, into the tree and the regular go command might not know they don't belong into the module. But, again, the important question here is what is or is not included in the module zip file.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Documentation help wanted NeedsFix The path to resolution is known, but the work has not been done.
Projects
None yet
Development

Successfully merging a pull request may close this issue.