cmd/go: go mod -vendor prunes non-package directories #26366

ainar-g · 2018-07-13T10:00:04Z

What version of Go are you using (`go version`)?

Does this issue reproduce with the latest release?

go version devel +8a33045 Fri Jul 13 03:53:00 2018 +0000 linux/amd64

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ainar/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/ainar/go"
GOPROXY=""
GORACE=""
GOROOT="/home/ainar/go/gotip"
GOTMPDIR=""
GOTOOLDIR="/home/ainar/go/gotip/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build865981061=/tmp/go-build -gno-record-gcc-switches"

What did you do?

#!/bin/sh
set -e
export GO111MODULE=on
mkdir foo
cd foo
cat <<EOF > main.go
package foo // import "foo"

import "github.com/gen2brain/aac-go/aacenc"

func f() { _ = aacenc.VoPidAacMdoule }
EOF
$GO mod -init
$GO mod -vendor
$GO build -getmode vendor

What did you expect to see?

Successful build.

What did you see instead?

# github.com/gen2brain/aac-go/aacenc
vendor/github.com/gen2brain/aac-go/aacenc/aacenc.go:4:19: fatal error: voAAC.h: No such file or directory
 //#include "voAAC.h"
                   ^
compilation terminated.

The problem is that go mod -vendor pruned the external/aacenc folder, which contains the C code needed to build this package.

The text was updated successfully, but these errors were encountered:

bcmills · 2018-07-13T15:29:32Z

The cause here seems to be that we prune out directories that are not Go packages.

There is no voAAC.h in github.com/gen2brain/aac-go/aacenc/, but I do see one in github.com/gen2brain/aac-go/aacenc/external/aacenc/include/.

kardianos · 2018-07-14T05:57:09Z

Side note from govendor. By default govendor only picks up pkgs referenced, but I had to add an option to grab the whole sub tree for this very reason.

Mostly cgo deps, but sometimes other required resources.

...

Right now I'm considering making a tool that can be run after go mod -vendor to trim unused packages. But I can't leave what isn't there.

myitcv · 2018-07-14T10:49:21Z

I wonder whether in the world of modules what some/most people are actually intending to do in this situation (namely where they think they want/need to go mod -vendor) is snapshot the current module dependencies as opposed to the packages. Doing so would seem to address all of these points?

So then a go mod -modvendor (command/flags to be 🚲-shed-ed) would create a directory (say ./modvendor) that could then be used by GOPROXY=file:///path/to/modvendor.

That said, I don't use go mod -vendor and likely never will. But I do maintain an append-only cache of the modules that my CI then uses via GOPROXY=file://...

ainar-g · 2018-07-14T11:39:44Z

I don't think anyone actually wants to vendor everything. There are two problems here, that I think should not be mixed:

Non-Go code dependencies vendoring. This is what this issue is about. Whether we like it or not, Go has CGo. And C dependencies are hard. Some approaches to solve this problem (excluding the painful choice of teaching the go tool about C dependencies... the horror):
- Let module authors declare their non-Go code dependencies in their go.mod. Something like
```
external "github.com/gen2brain/aac-go/aacenc/external/aacenc"
```
- Define and document a convention for such dependencies (e.g. external directory, similar how we have conventions for internal and testdata).
- Explicitly give up and document that CGo is fundamentally incompatible with go mod -vendor. After all, even if you do have the C code in your vendor, there are no guarantees that that C code doesn't have some dependencies of its own, so hoping that /vendor will save you from build failures is rather naïve.
Binary tool dependency vendoring. This may or may not be a problem Go modules want to solve. @rsc said it in x/vgo: Support depending on binary modules #24051.

It is not a goal to expand into "linux distribution" territory.

But then this, too, should be documented explicitly. It would be a shame though, as AFAIK other systems, like Ruby's Bundler and JS's NPM support some version of it.

kardianos · 2018-07-14T14:20:50Z

One way to solve the cgo problem is to require a dummy go file in the c folder that is then imported by another package that uses the c files with a "_" import.

I think this is only a problem because the c and h files live in a separate directory from the go files.

rsc · 2018-07-17T18:32:30Z

Sorry, but the model for go builds is that what's needed to build a package is in that directory, not subdirectories. The way gen2brain/aac-go is structured, modifying the C source code in the subdirectories will not trigger a rebuild. Instead your build will use stale cache entries. The C code needs to be moved up to the package directory to make it build properly, not to mention vendor properly.

If anyone wants to build a tool that adds to the vendor directory after go mod -vendor, you could do that by reading the modules.txt file and using commands like go list -f '{{.Dir}}' to find out where each package was copied from.

ainar-g · 2018-07-17T19:26:47Z

@rsc I see two opportunities for improvement here. Firstly, should we explicitly clarify in the cgo documentation that the go tool notices only changes in Go packages? Right now the wording doesn't mention other directories and local includes at all. Something like

<...> Any .h, .hh, .hpp, or .hxx files will not be compiled separately, but, if these header files are changed, the C and C++ files will be recompiled. Changes in files included from other directories will not trigger a rebuild. <...>

(Emphasis = added by me.)

Secondly, should such local includes be a vet warning? There is already -cgocall, why not -cgoinclude?

gopherbot · 2018-07-20T15:31:18Z

Change https://golang.org/cl/125297 mentions this issue: cmd/cgo: document that #including source files in subdirectories is a bad idea

… bad idea Suggested in discussion on #26366. Change-Id: Id9ad2e429a915f88b4c4b30fc415c722eebe0ea4 Reviewed-on: https://go-review.googlesource.com/125297 Reviewed-by: Ian Lance Taylor <iant@golang.org>

pkieltyka · 2018-08-27T17:51:54Z

this tool might be helpful to others: https://github.com/goware/modvendor — I wrote it to easily copy .c, .h, .s, .proto files from module pkgs into my local ./vendor. Feel free to improve :)

im-kulikov · 2018-09-10T23:06:26Z

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files 😔

myitcv · 2018-09-11T11:23:00Z

I've just added a "proposal" under #27618 (not marked as a proposal because the modules stuff is experimental, so more a discussion issue than anything). Apologies for the modvendor name clash @pkieltyka, I just spotted that.

nomad-software · 2018-09-24T17:47:11Z

I can't believe the proposed solution to this issue is to write another tool to finish what go mod vendor starts. Either write a tool to replace it or fix it so it copies everything. What is the reason for only vendoring half of the repository?

myitcv · 2018-09-24T18:21:53Z

@nomad-software Russ outlined a response to this in #26366 (comment).

If instead you want to "vendor" the module that will give you everything. That's the subject of #27618 and a proposed approach is outlined at https://github.com/myitcv/go-modules-by-example/blob/master/012_modvendor/README.md.

thepudds · 2018-09-25T03:54:44Z

@nomad-software this issue here was initially reported when the original reporter was using cgo.

Are also hitting this with cgo, vs. are you hitting this without using cgo?

If you are hitting this with cgo, and if we set aside vendoring for a moment, do you think you might be hitting problems even just during non-vendor-based building/recompiling due to the behavior described here:

https://golang.org/cmd/cgo/

When the Go tool sees that one or more Go files use the special import "C", it will look for other non-Go files in the directory and compile them as part of the Go package.
...
Note that changes to files in other directories do not cause the package to be recompiled, so all non-Go source code for the package should be stored in the package directory, not in subdirectories.

(It is worth reading the full description at that link, but wanted to include at least a snippet for convenience)

nomad-software · 2018-10-22T18:03:49Z

go mod vendor drops github.com/ethereum/go-ethereum/crypto/secp256k1/libsecp256k1 files

@im-kulikov I've written a tool to fully copy the entire directory of all dependencies into the vendor folder. You can find it here:

https://github.com/nomad-software/vend

Any suggestions to improve it are welcome, just open an issue there.

dolmen · 2022-02-20T16:37:13Z

Looking #26366 (comment), it seems that using a tree of C files will not work correctly.

But I still have to use code from a library that is organized as a tree. So I need to flatten that tree as files in my package directory.

Has anyone built a tool that will take a C file and resolve and inline all #include that are relative to a base directory? This is a subset of what a preprocessor does...

There is a rule for go build: all sources files of a given package must be in the directory of that package. Not in subdirectories. Which means that {#include "hidapi/mac/hid.c"} is not correct as hid.c is not in the package directory, so "go build" will not notice its changes and go mod -vendor will not work (golang/go#26366). So this patch creates 3 files (hidapi_linux.h, hidapi_windows.h, hidapi_mac.h) which are go-generated by internal/gen.go. The generator just inlines any includes from the libusb/ and hidapi/ directories. This means that the libusb/ and hidapi/ are not needed anymore for distribution (and for go mod -vendor): they are used only for the "go generate" phase.

To allow 'go mod -vendor' to take directories, we create dummy packages in each directory. Basically it does: for d in $(find internal -type d); do echo "package $(basename $d)" >"$d/pkg.go"; done A script is used to maintained those dummy package besides hidapi/libusb upgrades: internal/dummy-packages.sh See: - karalabe#31 - golang/go#26366

dolmen · 2022-02-20T22:34:44Z

Here is a workaround that may be useful for other projects:

For package github.com/dolmen-go/hid (which embeds C libraries libusb and hidapi) I've implemented an ad-hoc C preprocessor (triggered by go generate) that concatenates all required C files from the subdirectories and produces 3 .h files (one for each supported platform). The cgo source just includes the the local .h file corresponding to GOOS.

williamh · 2022-02-27T00:08:57Z

All,

I'm a packager on Gentoo, and I'm hitting this with https://git.sr.ht/~rjarry/aerc.git.

I am required to vendor to build packages under our package manager, and it doesn't look like that is going to change any time soon. Our package manager team sees allowing me to reach out to the network during a build as a massive security hole they don't want to open, and they tell me that having builds that are required to reach out to the network breaks building on unconnected systems.

So, what is the best way to deal with this issue?

Thanks,

William

williamh · 2022-02-27T04:17:36Z

I have packaged the vend tool from @nomad-software on Gentoo, thanks for writing it.

IMO this is a pretty significant issue since go mod vendor creates vendor directories that are unusable in some situations.

We will use vend for now, but please count me in as someone requesting that this discussion be restarted.

Thanks much for your time,

William

xuyang2 · 2022-03-30T00:58:20Z

@williamh
Is it acceptable to install athens as a private proxy for Go modules?
In this way, only athens needs to access the internet.

Remove psx_pthread_create() from libpsx - given the way -lpsx is linked this is not needed. Also, as pointed out by Lorenz Bauer, "go mod vendor" support was unable to vendor a copy of psx_syscall.h because it didn't reside in the same directory as the *.go code for the psx package. (General discussion golang/go#26366 .) Given that we can, avoid the use of a sub-directory in the libcap tree. Signed-off-by: Andrew G. Morgan <morgan@kernel.org>

acestronautical · 2022-05-11T22:15:12Z

Why is this closed? This is still a major issue. Go dep was able to handle this with "unused-packages = false" which means many cannot and will not switch over to go mod.

Use wkhtmltopdf for PDF support because it has the easiest emoji solution (via twemoji). heic2jpg.go is untested as it is largely copied from upstream, and I have a PR open to create an interface upstream: adrium/goheif#2 goheif also has issues vendoring because there is no Go code in some of the required C++ subdirectories: golang/go#26366. This should also be fixed upstream, but as a stopgap, we need to use the cp and chmod commands (vendor makefile target). * Enable output to PDF * Add support or jpegs in PDFs * Switch to wkhtmltopdf for emoji support * Implement copying attachments * Convert HEIC attachments to JPEG for PDF output * Update example to match new defaults * Add "attached" text to attachment references in outfiles * Update dependencies * Install wkhtmltopdf in Travis CI * Use xvfb in Travis CI

bcmills · 2023-03-21T18:52:46Z

Why is this closed?

See #26366 (comment).

myitcv added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. modules labels Jul 13, 2018

myitcv assigned rsc Jul 13, 2018

bcmills changed the title ~~cmd/go: go mod -vendor prunes C files~~ cmd/go: go mod -vendor prunes non-package directories Jul 13, 2018

ainar-g mentioned this issue Jul 14, 2018

x/vgo: Support depending on binary modules #24051

Closed

rsc closed this as completed Jul 17, 2018

rsc mentioned this issue Jul 20, 2018

x/tools/go/analysis/cmd/vet: add -cgoinclude check for #include of code in subdirectories #26506

Open

bcmills mentioned this issue Aug 2, 2018

cmd/go: build cache not updating when pkg-config files changed #24544

Closed

myitcv mentioned this issue Sep 11, 2018

cmd/go: add modvendor sub-command #27618

Open

myitcv mentioned this issue Sep 24, 2018

go mod vendor is not copying all files to the vendor folder #27832

Closed

myitcv mentioned this issue Sep 26, 2018

cmd/go: error when enable cgo and static link with build flag '-mod=vendor' #27667

Closed

sanguohot mentioned this issue Sep 27, 2018

crypto/secp256k1: libsecp256k1/include/secp256k1.h error when using go mod command with build flag '-mod=vendor' ethereum/go-ethereum#17700

Closed

fedepaol mentioned this issue Sep 27, 2018

goracle incompatible with gomods go-goracle/goracle#104

Closed

andrewchambers mentioned this issue Oct 10, 2018

Go module vendor - lib/sqlite3.h: No such file or directory bvinc/go-sqlite-lite#6

Closed

0xd61 mentioned this issue Feb 6, 2022

Add dummy files to preserve vendor directories inkyblackness/imgui-go#172

Merged

lindlof mentioned this issue Apr 8, 2022

Missing files when using go mod vendor 99designs/gqlgen#2081

Closed

mcardy mentioned this issue Apr 8, 2022

Support go mod vendor by ensuring that Cgo dependencies are vendored SolaceDev/pubsubplus-go-client#3

Merged

Marwes mentioned this issue May 12, 2022

refactor: Remove the need for the flux_test_gen files influxdata/flux#4729

Merged

rsc removed their assignment Jun 23, 2022

SteffenL mentioned this issue Jun 28, 2022

Custom WebView2Loader implementation webview/webview#783

Merged

7 tasks

tagatac mentioned this issue Jul 14, 2022

Enable export to PDF and copying of attachments tagatac/bagoup#11

Merged

tagatac mentioned this issue Jul 16, 2022

Improve HEIC conversion tagatac/bagoup#16

Open

flicaflow mentioned this issue Aug 8, 2022

Installation errors "go install" sqlc-dev/sqlc#1473

Closed

graywolf-at-work mentioned this issue Aug 25, 2022

Vendoring with go mod vendor leads to "fatal error: 'pg_query.h' file not found" pganalyze/pg_query_go#61

Open

davidcassany mentioned this issue Oct 6, 2022

Add vendor for obs integration rancher/elemental-operator#198

Merged

leycec mentioned this issue Oct 25, 2022

[Packaging] System package manager-friendly vendoring metafates/mangal#111

Closed

seankhliao mentioned this issue Nov 23, 2022

mod/vendor: broken support for git-submodules in vendoring mode #56920

Closed

asmaloney mentioned this issue Nov 26, 2022

go vendor does not work AllenDang/cimgui-go#34

Open

yann-soubeyrand mentioned this issue Dec 3, 2022

Error building without CGO go-git/go-git#624

Closed

tobuh mentioned this issue Dec 25, 2022

using simulator in other project does not fetch ms-tpm-20-ref google/go-tpm-tools#274

Closed

seankhliao mentioned this issue Dec 30, 2022

cmd/go: document that go mod vendor ignores directories that do not contain Go files #57529

Open

golang locked as resolved and limited conversation to collaborators Mar 21, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

cmd/go: go mod -vendor prunes non-package directories #26366

cmd/go: go mod -vendor prunes non-package directories #26366

ainar-g commented Jul 13, 2018

bcmills commented Jul 13, 2018

kardianos commented Jul 14, 2018

myitcv commented Jul 14, 2018

ainar-g commented Jul 14, 2018 •

edited

kardianos commented Jul 14, 2018 •

edited

rsc commented Jul 17, 2018

ainar-g commented Jul 17, 2018

gopherbot commented Jul 20, 2018

pkieltyka commented Aug 27, 2018

im-kulikov commented Sep 10, 2018

myitcv commented Sep 11, 2018

nomad-software commented Sep 24, 2018 •

edited

myitcv commented Sep 24, 2018

thepudds commented Sep 25, 2018

nomad-software commented Oct 22, 2018 •

edited

dolmen commented Feb 20, 2022 •

edited

dolmen commented Feb 20, 2022

williamh commented Feb 27, 2022

williamh commented Feb 27, 2022

xuyang2 commented Mar 30, 2022

acestronautical commented May 11, 2022

bcmills commented Mar 21, 2023

cmd/go: go mod -vendor prunes non-package directories #26366

cmd/go: go mod -vendor prunes non-package directories #26366

Comments

ainar-g commented Jul 13, 2018

What version of Go are you using (go version)?

Does this issue reproduce with the latest release?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

bcmills commented Jul 13, 2018

kardianos commented Jul 14, 2018

myitcv commented Jul 14, 2018

ainar-g commented Jul 14, 2018 • edited

kardianos commented Jul 14, 2018 • edited

rsc commented Jul 17, 2018

ainar-g commented Jul 17, 2018

gopherbot commented Jul 20, 2018

pkieltyka commented Aug 27, 2018

im-kulikov commented Sep 10, 2018

myitcv commented Sep 11, 2018

nomad-software commented Sep 24, 2018 • edited

myitcv commented Sep 24, 2018

thepudds commented Sep 25, 2018

nomad-software commented Oct 22, 2018 • edited

dolmen commented Feb 20, 2022 • edited

dolmen commented Feb 20, 2022

williamh commented Feb 27, 2022

williamh commented Feb 27, 2022

xuyang2 commented Mar 30, 2022

acestronautical commented May 11, 2022

bcmills commented Mar 21, 2023

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

ainar-g commented Jul 14, 2018 •

edited

kardianos commented Jul 14, 2018 •

edited

nomad-software commented Sep 24, 2018 •

edited

nomad-software commented Oct 22, 2018 •

edited

dolmen commented Feb 20, 2022 •

edited