I found a suggested workaround in some repository stating go <cmd> $(go list ./... | grep -v /vendor/) would solve this. Unfortunately, this does not work on Windows (no grep and so on). So how do I do this on Windows or is there maybe a flag or something I can use to ignore the vendor directory? I guess many people switching to vendor will encounter the same problem when testing the whole application at once.

My proposal for this is to add the flag -excludepkg to go test. The given packages should still be built with the other matching packages, but their tests would be skipped.
With that given, we can easy run all tests in directory omitting vendor directory completely:

go test ./... -excludepkg ./vendor...

Here is my CL for that: https://golang.org/cl/19972

CL https://golang.org/cl/19972 mentions this issue.

@minux yes, the packages are specified as import paths, so to exclude non- top-level vendored packages one could use -excludepkg ./...vendor... or -excludepkg ./vendor...,./.../vendor...

I find this syntax confusing and unfriendly. I would really prefer if
-excludepkg was not an option, but just did the right thing by default.

On Sat, Feb 27, 2016 at 8:05 PM, Dmitriy Dudkin notifications@github.com
wrote:

@minux https://github.com/minux yes, the packages are specified as
import paths, so to exclude non- top-level vendored packages one could use -excludepkg
./...vendor... or -excludepkg ./vendor...,./.../vendor...

—
Reply to this email directly or view it on GitHub
#14417 (comment).

I agree with @davecheney . Why not make a --all or --vendor option instead?

@davecheney @arekkas yeah, yesterday it looked better for me.
I think we can skip vendor tests by default and support -vendor option to enable them. I'll submit patch set for that.

https://golang.org/cl/19972
Uploaded patch set with the next changes:

• Ignore vendor tests by default
• Added -vendor option to include vendor tests

Well the issue is that things don't work any more as they used to / are expected to work. This is bad. Go is about reducing complexity and unifying the way we write code. This is why the standard lib is so awesome, powerfull and well designed.

Although this has been discussed in length, it is still something that is missing a cross platform, simple solution and i guarantee that it will be something where developers run into issues and that this sort of reports will come up time and time again.

I urge the maintainers to reconsider their point of view and find a viable, easy solution to cover the most basic thing in the toolchain: vendoring + testing.

----- Reply message -----
Von: "Minux Ma" notifications@github.com
An: "golang/go" go@noreply.github.com
Cc: "Aeneas" aeneas.rekkas@serlo.org
Betreff: [go] cmd/go: go test ./... fails on go 1.6 because dependencies are tested (#14417)
Datum: So., Feb. 28, 2016 04:13

This has been discussed at length on #11659,
and I don't feel comfortable to exclude vendor
packages from testing.

I proposed that we introduce negative package
patterns to cmd/go, but that was rejected.

Reply to this email directly or view it on GitHub:
#14417 (comment)

I like negative patterns/-excludepkg key better because it is more generally useful solution. But we can still go with skipping vendor tests by default as long as we will inform users about that. I think that printing something like ? vendor/pkg [skipped] is enough for someone to notice that vendor tests were not running and run tests with -vendor flag if they've wanted it.

If you don't want to test your vendor code, don't vendor it. (edit: it == the vendor tests)

If you only want to test your vendor code sometimes, vendor it with +build vendortest at the top of each yourdep_test.go file, so then to get those tests, you'll need to do:

$ go test -tags=vendortest ./...

If you don't want to test your vendor code, don't vendor it.

But I do need to vendor the code in order to have reliable builds. That's the whole point of vendoring, isn't it?

@arekkas, no, "it" == "your vendor code's test".

I'm really sorry, maybe I've got the terminology wrong, but I do not understand what you are saying :)

The point I'm trying to make is that vendored imports often are not managed by oneself (e.g. gorilla/mux, sqlx, httprouter, ...). Test break sometimes due to the maintainer's fault or due to configuration issues (e.g. missing environment variables). I want to have an option to make testing vendored imports optional, so go test ./... does not exit with an error.

If I understood correctly, I could add build flags to all *_test.go files inside of /vendor. But I do not have control over most of the dependencies, as seen in the example above: google.golang.org/cloud/internal/transport

I want my code to be test- and buildable in 2 years as well, this is why I am vendoring all imports.

Please don't take this as unwillingness or insult, I am trying to understand how to resolve the issue I described.

If you put them in your vendor directory, it seems you have control over whether they have +build tags in them.

I'm proposing that one of the vendoring tools (or a shell script) modify the vendor/**/*_test.go files to add +build vendortest.

Ah I see! Sounds good to me :)

@arekkas I agree with your concerns regarding CI test etc, @bradfitz and you have the same result in that tests on your build can include or exclude vendor code using a flag (see again https://go-review.googlesource.com/#/c/19972/) . Having said that, there will probably be some discussion around flagging any build that does not pass all tests including the vendored code. I think its good practice to write the *_test.go accordingly. It looks like 19972 is still open.

In my project I'm using git submodules to manage the contents of my vendor subdirectory. I know you guys don't necessarily care about / explicitly consider git submodules when you design stuff, but I've found it's a convenient way to manage my vendored packages, and smoothly update them to new levels if I need to. Modifying all the vendored packages' tests with the build flag doesn't seem to fit smoothly with git submodules. Say I want to update the level of one of my dependencies... I would have to manage my own build flag patch over each repository, unapply my patch, merge in the new changes, re-apply my patch on top, and then commit this state of the vendored submodule. This isn't really practical for me and I probably won't use this workaround.

I already have the go test -race $(go list ./... | grep -v /vendor/) trick scripted everywhere. I personally like the approach of the CL initially given here. I have some vendored dependencies that work well in my projects but have busted tests (namely their test dependencies). You could make the argument that I shouldn't use such packages, but it's hard to find good alternatives, it works well, and I don't have time to roll my own right now on account of their tests, or to fix their tests.

I'm just some random irrelevant dude but personally I think recommending not vendoring tests isn't super fair.

I've found personally that using git submodules is the nicest way to include the deps as well, but then you're stuck with their tests as well. Even if I wasn't using git submodules I don't really want to write scripts to remove the tests etc.

I know everyone is against renaming ./vendor. I would rather see ./vendor and ./_vendor supported than have to deal with the tainted user experience. Go already has this ad-hoc notion of ./vendor now, so I don't see why the tooling shouldn't be aware of it as well.

Go added go generate to get rid of Makefiles, now I need Makefiles just to use the toolchain. For me one of the biggest selling points of Go, and one I advocate is that it (usually) encourages sameness, this breaks that convention.

I get it now, @bradfitz . Took me a few days but I finally broke the box of thought I was in. ;) I always thought of the vendor directory as something immutable, similar to node_modules. But if one doesn't see it as such (and it isn't, because we're commiting the stuff, right?), it is actually task of the vendoring tools, respectively godep or glide, to have an option for removing or tagging tests.

Thanks for your time and explanation. Maybe this should be documented somewhere, so people like me don't get all confused. :)

Closing because you're right. And cudos for not giving in, despite the lengthy discussions! This is why go rocks.

@arekkas this isn't possible if you use submodules (which is much nicer than checking all the source in), and I would argue that since Go has a built-in notion of ./vendor that it is first-class in a sense, just like node_modules.

@tj, whether git submodules are nicer than checking all the source is very much subjective. There are pros & cons either way, and people will weigh those differently. But if you prefer submodules yourself, you have to weigh that against the inability to alter what you check in with +build tags and the inability to use go test ./... indiscriminately. You'll just have to specify which tests you want to run yourself, or configure your codereview or CI system or editor to run the tests you actually care about.

FWIW: godep hasn't saved the dependency's test files since september of last year (~v9), specifically because of this. I also fixed a bug recently where godep was vendoring more than it should have (because of old assumptions in the code). Please see the linked issue above.

I'm using the vendor directory to avoid problems similar to leftpad, and I assume that this was its intention to begin with. The problem I'm seeing here is that if my assumption is correct, the vendor directory should be populated by modules that are from external sources and considered to have already been tested and suitable for my use. So to include them in the './...' pattern simply prolongs the feedback loop for unit tests, which is undesirable, and in the case of 'vendor' probably completely unnecessary.

And I don't think having to augment the installed vendor packages is a clean solution. But maybe I don't understand Go very well.

And I don't think having to augment the installed vendor packages is a clean solution. But maybe I don't understand Go very well.

The idea is, that if you vendor things, you should also know what they do and if they work. If you want to skip that test, you have to do so explicitly.

Just remember, windows users (in general) do not have access to tools like
grep, so the workaround proposed above excludes a large percentage of Go's
userbase.

On Wed, 7 Sep 2016, 16:58 Aeneas notifications@github.com wrote:

And I don't think having to augment the installed vendor packages is a
clean solution. But maybe I don't understand Go very well.

The idea is, that if you vendor things, you should also know what they do
and if they work. If you want to skip that test, you have to do so
explicitly.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#14417 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AAAcA-L2H1Vy0Mf9t0jk9Je2XU20lMLbks5qnmCbgaJpZM4HeXgt
.

Just remember, windows users (in general) do not have access to tools like grep, so the workaround proposed above excludes a large percentage of Go's userbase.

💯 - I have to use git bash to test go on windows or delete vendor contents. A flag like -novendor would probably already do it

One thing to note here is, that tests fail if vendored dependencies ship or use internal packages