Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: directories named vendor are ignored in //go:embed #45744

Open
jakelucas opened this issue Apr 24, 2021 · 4 comments
Open

cmd/go: directories named vendor are ignored in //go:embed #45744

jakelucas opened this issue Apr 24, 2021 · 4 comments
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone

Comments

@jakelucas
Copy link

What version of Go are you using (go version)?

$ go version
go version go1.16.3 windows/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
set GO111MODULE=
set GOARCH=amd64
set GOBIN=D:\projects\go\bin
set GOCACHE=D:\devenv\go\1.16.3\..\cache
set GOENV=C:\Users\\AppData\Roaming\go\env
set GOEXE=.exe
set GOFLAGS=
set GOHOSTARCH=amd64
set GOHOSTOS=windows
set GOINSECURE=
set GOMODCACHE=D:\projects\go\pkg\mod
set GONOPROXY=
set GONOSUMDB=
set GOOS=windows
set GOPATH=D:\projects\go
set GOPRIVATE=
set GOPROXY=https://proxy.golang.org,direct
set GOROOT=D:\devenv\go\1.16.3
set GOSUMDB=sum.golang.org
set GOTMPDIR=
set GOTOOLDIR=D:\devenv\go\1.16.3\pkg\tool\windows_amd64
set GOVCS=
set GOVERSION=go1.16.3
set GCCGO=gccgo
set AR=ar
set CC=gcc
set CXX=g++
set CGO_ENABLED=1
set GOMOD=NUL
set CGO_CFLAGS=-g -O2
set CGO_CPPFLAGS=
set CGO_CXXFLAGS=-g -O2
set CGO_FFLAGS=-g -O2
set CGO_LDFLAGS=-g -O2
set PKG_CONFIG=pkg-config
set GOGCCFLAGS=-m64 -mthreads -fmessage-length=0 -fdebug-prefix-map=C:\Users\~1\AppData\Local\Temp\go-build3923578036=/tmp/go-build -gno-record-gcc-switches

What did you do?

Used //go:embed <dir> to recursively embed static website assets into an embed.FS variable, and then ran go install <repo> to install the binary into gobin.

What did you expect to see?

All assets embedded into the embed.FS variable in the final binary in gobin.

What did you see instead?

Only some assets were embedded in the embed.FS variable in the final binary in gobin.

The problem comes from having folders named vendor within the directory that gets embedded.

If you create a project that embeds a directory, for example //go:embed static, which includes some sub-directory named vendor, for example static/vendor/style.css, then the embed behaviour is different depending on how you install the project into gobin.

If you have the entire project source locally and you run go install within the project then the vendor directory is correctly included and embedded into the final binary.

If you push the project to somewhere like GitHub and then try to install via go install <repo> or go get <repo>, then all files are correctly embedded according to the documentation except for the entire vendor folder, which is ignored. You can work around this by setting GO111MODULE=auto. If this is set then the vendor directory is no longer ignored and is correctly embedded in the final binary.

I believe this is just an unintended side-effect of go get/go install and how they deal with vendored dependencies. If it is actually intended then it at the very least needs to be documented and made consistent across the different methods of installation.

@seankhliao seankhliao changed the title Directories named vendor are ignored in //go:embed cmd/go: directories named vendor are ignored in //go:embed Apr 24, 2021
@cherrymui cherrymui added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Apr 26, 2021
@cherrymui cherrymui added this to the Backlog milestone Apr 26, 2021
@cherrymui
Copy link
Member

cc @bcmills @rsc

@rsc
Copy link
Contributor

rsc commented May 4, 2021

We can only embed files that are saved in the packaging of the module, and directories named vendor are excluded from the module. Therefore they need to be excluded from the embed as well.

@patrickbsf
Copy link

patrickbsf commented Aug 27, 2021

Could you just make sure it's at least mentioned in the go:embed documentation at least? This isn't obvious.

@bcmills
Copy link
Contributor

bcmills commented Nov 1, 2021

This is currently mentioned in the embed package does, although it could stand to be clearer for vendor in particular.

Today (https://pkg.go.dev/embed@go1.17.2) it says:

Patterns must not match files outside the package's module, such as ‘.git/*’ or symbolic links. Matches for empty directories are ignored. After that, each pattern in a //go:embed line must match at least one file or non-empty directory.

Perhaps we are also missing a check to cause the build to error out if the pattern does match files in the vendor directory? That should be an explicit error — it shouldn't implicitly drop files that match, although things get a bit tricky when we consider glob patterns (which should just stop at module boundaries without erroring out).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Development

No branches or pull requests

5 participants