Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: "go test" cannot resolve external dependencies within src directory #18606

Closed
askthedragon opened this issue Jan 11, 2017 · 45 comments
Closed
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.

Comments

@askthedragon
Copy link

askthedragon commented Jan 11, 2017

Description

When running 'go test' from a path that conforms to normal golang directory structure (/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder), build dependencies aren't getting resolved.

I am able to successfully run 'go test' from a directory within my $GOPATH as long as it isn't within the src directory (/Users/builduser/Projects/golang/cb-event-forwarder).

When running 'go test -x -v' from the conformant (/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder) directory the pkg dependency directory is not set when attempting to compile:

/usr/local/Cellar/go/1.7.4_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/github.com/carbonblack/cb-event-forwarder/_test/github.com/carbonblack/cb-event-forwarder.a -trimpath $WORK -p main -complete -buildid bf76e4715ca73e0af88f96d25285fab208f5e2fb -D _/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder -I $WORK -I "" -pack ./amqp.go ./bundled_output.go ./cb_api.go ./config.go ./file_output.go ./http_behavior.go ./json_message_processor.go ./main.go ./net_output.go ./pb_message_processor.go ./s3_behavior.go ./syslog_output.go ./utils.go ./event_processing_test.go ./json_message_processor_test.go ./leef_encoder_test.go ./leef_output_test.go

Notice the -I "" -pack . I can only deduce this is the issue when comparing to running 'go test' from a directory path (/Users/builduser/Projects/golang/cb-event-forwarder):

/usr/local/Cellar/go/1.7.4_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/_/Users/builduser/Projects/golang/cb-event-forwarder/_test/_/Users/builduser/Projects/golang/cb-event-forwarder.a -trimpath $WORK -p main -complete -buildid bf76e4715ca73e0af88f96d25285fab208f5e2fb -D _/Users/builduser/Projects/golang/cb-event-forwarder -I $WORK -I /Users/builduser/Projects/golang/pkg/darwin_amd64 -pack ./amqp.go ./bundled_output.go ./cb_api.go ./config.go ./file_output.go ./http_behavior.go ./json_message_processor.go ./main.go ./net_output.go ./pb_message_processor.go ./s3_behavior.go ./syslog_output.go ./utils.go ./event_processing_test.go ./json_message_processor_test.go ./leef_encoder_test.go ./leef_output_test.go

TLDR 'go test' works when within $GOPATH as long as it isn't within the src directory, otherwise it can't find dependencies. I am one of the maintainers of the golang code in question. Thank you for your time.

What version of Go are you using (go version)?

go version go1.7.4 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/builduser/Projects/golang"
GORACE=""
GOROOT="/usr/local/Cellar/go/1.7.4_1/libexec"
GOTOOLDIR="/usr/local/Cellar/go/1.7.4_1/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/s0/9xk9c2dd49jfh5rgsq36h4ym0000gn/T/go-build632545478=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

$ go get github.com/carbonblack/cb-event-forwarder
$ go get -u github.com/golang/protobuf/proto
$ go get -u github.com/golang/protobuf/protoc-gen-go
$ go generate ./...
$ go get ./...
$ go test

What did you expect to see?

$ go test
PASS
ok    _/Users/builduser/Projects/golang/cb-event-forwarder  2.733s

What did you see instead?

$ go test
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
FAIL  github.com/carbonblack/cb-event-forwarder [build failed]
@davecheney
Copy link
Contributor

The underscore in the package name reported by the go tool says you haven't set GOPATH correctly.

Sorry, you have to live in GOPATH's world if you want to use the go tool.

@askthedragon
Copy link
Author

I thought my GOPATH is set. My go env says it is set to "GOPATH="/Users/builduser/Projects/golang"

@davecheney
Copy link
Contributor

The fact that go get worked says to me that you to have GOPATH set, but you're trying to use the go tool to build code that is not inside your GOPATH. Sorry, perhaps this was the issue reported.

Sorry, what I think you're trying to do isn't supported, the go tool will try to resolve each package relative to what it can find inside GOPATH and will build your code against the copy inside GOPATH.

@askthedragon
Copy link
Author

All dependencies are within my GOPATH.

$ pwd
/Users/builduser/Projects/golang/src
$ tree -L 3
.
├── github.com
│   ├── RackSec
│   │   └── srslog
│   ├── askthedragon
│   │   └── get-feed-title
│   ├── aws
│   │   └── aws-sdk-go
│   ├── carbonblack
│   │   ├── cb-event-forwarder
│   │   └── cbforwarder
│   ├── daviddengcn
│   │   └── go-colortext
│   ├── go-ini
│   │   └── ini
│   ├── golang
│   │   └── protobuf
│   ├── hashicorp
│   │   └── go-version
│   ├── jmespath
│   │   └── go-jmespath
│   ├── kardianos
│   │   └── govendor
│   ├── mattn
│   │   ├── gom
│   │   └── gover
│   ├── mohae
│   │   └── customjson
│   ├── streadway
│   │   └── amqp
│   ├── vaughan0
│   │   └── go-ini
│   └── zvelo
│       └── ttlru

Notice that the dependency (github.com/streadway/amqp) that couldn't be found is in the tree above, which is within my GOPATH (/Users/builduser/Projects/golang) .

@davecheney
Copy link
Contributor

Your code doesn't build with a clean checkout

lucky(~/src/github.com/carbonblack/cb-event-forwarder) % go test
# github.com/carbonblack/cb-event-forwarder
leef_encoder_test.go:4:2: local import "./leef" in non-local package
FAIL    github.com/carbonblack/cb-event-forwarder [setup failed]

@davecheney
Copy link
Contributor

Quick test, have you used symlinks inside your GOPATH?

@askthedragon
Copy link
Author

I have not used any symlinks.

@askthedragon
Copy link
Author

@davecheney what version of golang? I want to reproduce your output here locally.

@davecheney
Copy link
Contributor

The repo you provided doesn't build or test cleanly.

% go get github.com/carbonblack/cb-event-forwarder
go test # github.com/carbonblack/cb-event-forwarder
/home/dfc/src/github.com/carbonblack/cb-event-forwarder/pb_message_processor.go:19: undefined: sensor_events.CbEventMsg
/home/dfc/src/github.com/carbonblack/cb-event-forwarder/pb_message_processor.go:33: undefined: sensor_events.CbEventMsg
deadwood(~/src/stash.atlassian.com/scm/kube/kubetoken/internal/cert) % go test github.com/carbonblack/cb-event-forwarder
# github.com/carbonblack/cb-event-forwarder
/home/dfc/src/github.com/carbonblack/cb-event-forwarder/leef_encoder_test.go:4:2: local import "./leef" in non-local package
FAIL    github.com/carbonblack/cb-event-forwarder [setup failed]

Are you sure you're working with the right copy of your source?

@askthedragon
Copy link
Author

just did a clean checkout on master and getting the same error.

$ go test
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
FAIL	github.com/carbonblack/cb-event-forwarder [build failed]

@askthedragon
Copy link
Author

askthedragon commented Jan 11, 2017

ok just did a full build on a clean vm.

$ go get github.com/carbonblack/cb-event-forwarder
$ go get -u github.com/golang/protobuf/proto
$ go get -u github.com/golang/protobuf/protoc-gen-go
$ go generate ./...
$ go get ./...
$ go test
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
FAIL	github.com/carbonblack/cb-event-forwarder [build failed]

@davecheney
Copy link
Contributor

davecheney commented Jan 11, 2017 via email

@askthedragon
Copy link
Author

I can reproduce on Linux and Mac. The Linux VM was clean. Any ideas on what could the problem possibly be?

I have updated build instructions.

@davecheney
Copy link
Contributor

I can reproduce on Linux and Mac. The Linux VM was clean. Any ideas on what could the problem possibly be?

My best guess is GOPATH is not set correctly, or your source is not within GOPATH.

@askthedragon
Copy link
Author

askthedragon commented Jan 11, 2017

Current Directory:

$ pwd
/home/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder

Here is the compile with the pkg directory missing as well on linux amd64:

/usr/local/go/pkg/tool/linux_amd64/compile -o $WORK/github.com/carbonblack/cb-event-forwarder/_test/github.com/carbonblack/cb-event-forwarder.a -trimpath $WORK -p main -complete -buildid 23cf5c6f9652ae5bee722b1913b83c0fb85e982a -D _/home/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder -I $WORK -I "" -pack ./amqp.go ./bundled_output.go ./cb_api.go ./config.go ./file_output.go ./http_behavior.go ./json_message_processor.go ./main.go ./net_output.go ./pb_message_processor.go ./s3_behavior.go ./syslog_output.go ./utils.go ./event_processing_test.go ./json_message_processor_test.go ./leef_encoder_test.go ./leef_output_test.go
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
$WORK/github.com/carbonblack/cb-event-forwarder/_test/cb-event-forwarder.test -test.v=true
FAIL	github.com/carbonblack/cb-event-forwarder [build failed]

Here is go env:

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/builduser/Projects/golang/"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
CC="gcc"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build328209578=/tmp/go-build -gno-record-gcc-switches"
CXX="g++"
CGO_ENABLED="1"

go version:

go version go1.7.4 linux/amd64

@askthedragon
Copy link
Author

askthedragon commented Jan 11, 2017

I just tested golang version 1.6.4 on linux and everything works fine. So there might be some regression in 1.7 that is causing 'go test' to not see these dependencies. Here is the go version that worked for me.

$ go version

go version go1.6.4 linux/amd64

This is probably why this issue popped up for us, since my colleague and I both recently upgraded to 1.7.4

@davecheney
Copy link
Contributor

That error is coming from cmd/compile, not the go tool. What's

ls -al $GOPATH/pkg/linux_amd64/github.com/streadway/

print ?

If you rm -rf $GOPATH/pkg then it will probably work. Normally the go tool should rebuild any package that doesn't match the current compiler, but in this case it doesn't appear to have done so.

@davecheney
Copy link
Contributor

What does

ls -al $GOPATH/src/github.com/streadway/

print ?

@askthedragon
Copy link
Author

$ ls -al $GOPATH/src/github.com/streadway/
total 3
drwxr-xr-x 1 501 dialout  102 Dec 18 21:40 .
drwxr-xr-x 1 501 dialout  612 Dec 28 12:51 ..
drwxr-xr-x 1 501 dialout 1360 Dec 18 21:40 amqp

@davecheney
Copy link
Contributor

sorry

ls -al $GOPATH/src/github.com/streadway/amqp

@askthedragon
Copy link
Author

$ ls -al $GOPATH/src/github.com/streadway/amqp
total 302
drwxr-xr-x 1 501 dialout  1360 Dec 18 21:40 .
drwxr-xr-x 1 501 dialout   102 Dec 18 21:40 ..
-rw-r--r-- 1 501 dialout  2195 Dec 18 21:40 allocator.go
-rw-r--r-- 1 501 dialout  2147 Dec 18 21:40 allocator_test.go
-rw-r--r-- 1 501 dialout  1203 Dec 18 21:40 auth.go
-rwxr-xr-x 1 501 dialout  2987 Dec 18 21:40 certs.sh
-rw-r--r-- 1 501 dialout 47212 Dec 18 21:40 channel.go
-rw-r--r-- 1 501 dialout 13725 Dec 18 21:40 client_test.go
-rw-r--r-- 1 501 dialout  2014 Dec 18 21:40 confirms.go
-rw-r--r-- 1 501 dialout  2228 Dec 18 21:40 confirms_test.go
-rw-r--r-- 1 501 dialout 21701 Dec 18 21:40 connection.go
-rw-r--r-- 1 501 dialout  2071 Dec 18 21:40 connection_test.go
-rw-r--r-- 1 501 dialout  2316 Dec 18 21:40 consumers.go
-rw-r--r-- 1 501 dialout  1190 Dec 18 21:40 CONTRIBUTING.md
-rw-r--r-- 1 501 dialout  5969 Dec 18 21:40 delivery.go
-rw-r--r-- 1 501 dialout   929 Dec 18 21:40 delivery_test.go
-rw-r--r-- 1 501 dialout  4925 Dec 18 21:40 doc.go
drwxr-xr-x 1 501 dialout   170 Dec 18 21:40 _examples
-rw-r--r-- 1 501 dialout 11757 Dec 18 21:40 examples_test.go
-rw-r--r-- 1 501 dialout   238 Dec 18 21:40 fuzz.go
-rwxr-xr-x 1 501 dialout    89 Dec 18 21:40 gen.sh
drwxr-xr-x 1 501 dialout   442 Dec 28 12:52 .git
-rw-r--r-- 1 501 dialout   100 Dec 18 21:40 .gitignore
-rw-r--r-- 1 501 dialout 45352 Dec 18 21:40 integration_test.go
-rw-r--r-- 1 501 dialout  1303 Dec 18 21:40 LICENSE
-rwxr-xr-x 1 501 dialout   948 Dec 18 21:40 pre-commit
-rw-r--r-- 1 501 dialout  9442 Dec 18 21:40 read.go
-rw-r--r-- 1 501 dialout  3727 Dec 18 21:40 README.md
-rw-r--r-- 1 501 dialout   529 Dec 18 21:40 read_test.go
-rw-r--r-- 1 501 dialout  2875 Dec 18 21:40 reconnect_test.go
-rw-r--r-- 1 501 dialout  2279 Dec 18 21:40 return.go
-rw-r--r-- 1 501 dialout  1575 Dec 18 21:40 shared_test.go
drwxr-xr-x 1 501 dialout   136 Dec 18 21:40 spec
-rw-r--r-- 1 501 dialout 59236 Dec 18 21:40 spec091.go
-rw-r--r-- 1 501 dialout  8712 Dec 18 21:40 tls_test.go
-rw-r--r-- 1 501 dialout   254 Dec 18 21:40 .travis.yml
-rw-r--r-- 1 501 dialout 14676 Dec 18 21:40 types.go
-rw-r--r-- 1 501 dialout  3713 Dec 18 21:40 uri.go
-rw-r--r-- 1 501 dialout  7334 Dec 18 21:40 uri_test.go
-rw-r--r-- 1 501 dialout  8530 Dec 18 21:40 write.go

@davecheney
Copy link
Contributor

ls -al $GOPATH/pkg/linux_amd64/github.com/streadway/

@askthedragon
Copy link
Author

so i did a rm -rf $GOPATH/pkg and go test now works

@askthedragon
Copy link
Author

$ ls -al $GOPATH/pkg/linux_amd64/github.com/streadway/
ls: cannot access '/home/builduser/Projects/golang//pkg/linux_amd64/github.com/streadway/': No such file or directory

@askthedragon
Copy link
Author

so does this mean prior to doing go test, we need to delete all the packages? Or are we doing something wrong that has changed in 1.7?

@davecheney
Copy link
Contributor

davecheney commented Jan 11, 2017

Ok, here's my best guess for what happed.
$GOPATH/pkg/linux_amd64/github.com/streadway/amqp.a was newer than any of the source files, so it was not scheduled for a rebuild by the go tool.
The compiler opens that file to read the symbols and finds it isn't the right version and rejects it.
The go tool should have scheduled amqp.a for rebuild, but it didn't.
The compiler should have given a better error, but it didn't.

Recommendation: use go install not go build, and you won't have issues like these in the future.

@askthedragon
Copy link
Author

so always do a go install prior to doing go test?

@davecheney
Copy link
Contributor

davecheney commented Jan 11, 2017 via email

@askthedragon
Copy link
Author

When I use go install on that amqp package I still get the same error. Basically I don't think the go tool is finding my pkg directory at all.

@davecheney
Copy link
Contributor

davecheney commented Jan 11, 2017 via email

@askthedragon
Copy link
Author

which command do you want? go install on the amqp package or a go test with the error?

@askthedragon
Copy link
Author

Here is the full go test -x -v

buildusers-MacBook-Pro:cb-event-forwarder builduser$ go test -x -v
WORK=/var/folders/s0/9xk9c2dd49jfh5rgsq36h4ym0000gn/T/go-build225381887
mkdir -p $WORK/github.com/carbonblack/cb-event-forwarder/_test/github.com/carbonblack/
mkdir -p $WORK/_/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/leef/_obj/
mkdir -p $WORK/_/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/
cd /Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/leef
/usr/local/Cellar/go/1.7.4_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/_/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/leef.a -trimpath $WORK -p _/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/leef -complete -buildid d934e1fb1a3c9bca60fe0bc6639f2b06d126df5c -D _/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder/leef -I $WORK -pack ./leef_encoder.go
mkdir -p $WORK/github.com/carbonblack/cb-event-forwarder/_test/_obj_test/
cd /Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder
/usr/local/Cellar/go/1.7.4_1/libexec/pkg/tool/darwin_amd64/compile -o $WORK/github.com/carbonblack/cb-event-forwarder/_test/github.com/carbonblack/cb-event-forwarder.a -trimpath $WORK -p main -complete -buildid bf76e4715ca73e0af88f96d25285fab208f5e2fb -D _/Users/builduser/Projects/golang/src/github.com/carbonblack/cb-event-forwarder -I $WORK -I "" -pack ./amqp.go ./bundled_output.go ./cb_api.go ./config.go ./file_output.go ./http_behavior.go ./json_message_processor.go ./main.go ./net_output.go ./pb_message_processor.go ./s3_behavior.go ./syslog_output.go ./utils.go ./event_processing_test.go ./json_message_processor_test.go ./leef_encoder_test.go ./leef_output_test.go
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
$WORK/github.com/carbonblack/cb-event-forwarder/_test/cb-event-forwarder.test -test.v=true
FAIL	github.com/carbonblack/cb-event-forwarder [build failed]

Notice how the pkg directory is missing in the compile flags

@askthedragon
Copy link
Author

Want to make sure I'm not doing anything wrong, but here is our full build instructions that I perform prior to running go test:

go get -u github.com/golang/protobuf/proto
go get -u github.com/golang/protobuf/protoc-gen-go
go generate ./...
go get ./...

@davecheney
Copy link
Contributor

Notice how the pkg directory is missing in the compile flags

Yup, something is wrong with your environment.

@davecheney
Copy link
Contributor

@askthedragon
Copy link
Author

I'm at a loss since everything compiles and tests fine with 1.6.4. In the morning I'll try your suggestion. Thank you for your time and I'll report back with updates.

@davecheney
Copy link
Contributor

davecheney commented Jan 11, 2017 via email

@askthedragon
Copy link
Author

Yea I made sure I started fresh.

@askthedragon
Copy link
Author

same error after deleting line 4 in leef_encoder_test.go:

$ go test
# github.com/carbonblack/cb-event-forwarder
./amqp.go:5: can't find import: "github.com/streadway/amqp"
FAIL	github.com/carbonblack/cb-event-forwarder [build failed]

@askthedragon
Copy link
Author

Also going to add that go build builds successfully

@rakyll
Copy link
Contributor

rakyll commented Jan 11, 2017

With the tip, I experience the following. After I fixed the the local import, go test succeeded. Is there a possible for you to try it with Go 1.8 beta?

$ go get github.com/carbonblack/cb-event-forwarder
# github.com/carbonblack/cb-event-forwarder
../src/github.com/carbonblack/cb-event-forwarder/pb_message_processor.go:19: undefined: sensor_events.CbEventMsg
../src/github.com/carbonblack/cb-event-forwarder/pb_message_processor.go:33: undefined: sensor_events.CbEventMsg
$ go test
# github.com/carbonblack/cb-event-forwarder
leef_encoder_test.go:4:2: local import "./leef" in non-local package
FAIL	github.com/carbonblack/cb-event-forwarder [setup failed]

@rakyll
Copy link
Contributor

rakyll commented Jan 11, 2017

Use the following commands to download and use the 1.8 beta:

$ go get golang.org/x/build/version/go1.8beta2
$ go1.8beta 2 download
$ go1.8beta test

I believe this is a build issue with an old installed library, wondering if 1.8 handles it already.

@rakyll rakyll added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Jan 11, 2017
@askthedragon
Copy link
Author

go 1.8 beta2 seems to have fixed my external dependency issue that was present in 1.7.4. This issue can be marked as closed as it has apparently been fixed in 1.8.

$ go1.8beta2 test
# github.com/carbonblack/cb-event-forwarder
leef_output_test.go:4:2: local import "./leef" in non-local package
FAIL	github.com/carbonblack/cb-event-forwarder [setup failed]

@askthedragon
Copy link
Author

Thanks all for your time

@cespare cespare closed this as completed Jan 11, 2017
@rakyll
Copy link
Contributor

rakyll commented Jan 11, 2017

so always do a go install prior to doing go test?

No, please report if you see any weird behavior.

@golang golang locked and limited conversation to collaborators Jan 11, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

5 participants