Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

cmd/go: go get -d -u on AWS code commit repository fails #32166

Closed
antgustech opened this issue May 21, 2019 · 8 comments
Closed

cmd/go: go get -d -u on AWS code commit repository fails #32166

antgustech opened this issue May 21, 2019 · 8 comments
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone

Comments

@antgustech
Copy link

antgustech commented May 21, 2019

What version of Go are you using (go version)?

$ go version 12.5

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/CORPUSERS/xxx/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/CORPUSERS/xxx/projects/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=

What did you do?

Trying to run go get on a repository hosted on code commit:

go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo

What did you expect to see?

The repository cloned and dependencies installed.

What did you see instead?

package git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo: unrecognized import path "git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo" (parse https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo?go-get=1: no go-import meta tags ())

My machine is properly setup with SSH keys so git clone ssh://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo works fine. Also take note that my end goal is to the the build pipeline on AWS Code build run the dockerfile in the end but as of now it gives the same result as running it locally.

@AlexRouSg
Copy link
Contributor

Try go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo.git

See https://golang.org/cmd/go/#hdr-Remote_import_paths

@antgustech
Copy link
Author

antgustech commented May 21, 2019

Try go get -d -u git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo.git

See https://golang.org/cmd/go/#hdr-Remote_import_paths

Adding .git to the end gave me another problem, so at least some progress. :)

package git-codecommit.eu-west-1.amazonaws.com/v1/repos/battery-historian.git: cannot download, git://git-codecommit.eu-west-1.amazonaws.com/v1/repos/myRepo uses insecure protocol

Isn't go get using HTTPS by default? How can that be insercure? If I add either SSH or HTTPS as the protocol, it fails sating the import path is malformed. I've seen solutions on how to specify using SSH for github but not understood how to do that for Code commit. I've tried adding


[url "git@git-codecommit.eu-west-1.amazonaws.com:"]
	insteadOf = https://git-codecommit.eu-west-1.amazonaws.com/v1/repos/

To my .gitconfig file but it did not help.

@antgustech antgustech changed the title go get -d -u code commit repository fail go get -d -u on AWS code commit repository fails May 21, 2019
@thepudds
Copy link
Contributor

Perhaps not directly related, but #25792 is older AWS CodeCommit vgo bug, and includes final comment:

It's fixed with one small detail.

Package imports as git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git require that packages go.mod contains module git-codecommit.us-east-1.amazonaws.com/v1/repos/return2.git.

@baxtex Probably worth at least skimming that bug.

@bcmills bcmills changed the title go get -d -u on AWS code commit repository fails cmd/go: go get -d -u on AWS code commit repository fails May 21, 2019
@bcmills bcmills added modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. labels May 21, 2019
@bcmills bcmills added this to the Go1.13 milestone May 21, 2019
@bcmills
Copy link
Contributor

bcmills commented May 22, 2019

@baxtex, the go command probes .git paths using all known protocol schemes for the VCS that meet security requirements (see here).

Can you git clone or git ls-remote the https URL for the path?

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label May 22, 2019
@antgustech
Copy link
Author

After some more research, I discovered that I probably don't need to do go get/git clone at all. As I'm running this on AWS code build, it already has access to the repository. So I realized I could use the COPY command instead. Here is my updated dockerfile:

#Use latest golang version 
FROM golang:latest 
 
MAINTAINER Me 
 
#Install Java 
RUN apt-get -y update && apt-get install -y openjdk-8-jre-headless 

#Get the battery historian code 
WORKDIR /go/src/codecommit/battery-historian 
COPY . /go/src/codecommit/battery-historian 
 
RUN go get ./... 
#RUN go get -d -u 
RUN go run setup.go 
 
#Open default port 
EXPOSE 8888 
 
#Run the start file for battery historian in a container 
CMD go run cmd/battery-historian/battery-historian.go --port 8888 --name my-battery-historian 

As you can see, I use copy to copy in my own battery historian repository instead of using go get. Do I have to do something more? Because when I study my logs, it looks like it cannot find all my dependencies:


Step 7/9 : RUN go run setup.go 
 ---> Running in f1735a0eb865 
 
Generating JS runfiles... 
Couldn't generate runfile: failed to run command "python /go/src/codecommit/battery-historian/third_party/closure-library/closure/bin/build/depswriter.py --root=/go/src/codecommit/battery-historian/third_party/closure-library/closure/goog --root_with_prefix=js ../../../../js": 
  exit status 2 
  python: can't open file '/go/src/codecommit/battery-historian/third_party/closure-library/closure/bin/build/depswriter.py': [Errno 2] No such file or directory 

For reference, the repo I got on Code commit is a clone of googles battery historian: https://github.com/google/battery-historian

@agnivade
Copy link
Contributor

agnivade commented Jun 8, 2019

@baxtex - Have you manually checked that third_party/closure-library/closure/bin/build/depswriter.py exists ?

Could you use rebuild=true and run docker build again ?

And in any case, this does not seem to be a Go issue anymore as you are just copying files.

@agnivade agnivade added WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. and removed WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Jun 8, 2019
@gopherbot
Copy link

Timed out in state WaitingForInfo. Closing.

(I am just a bot, though. Please speak up if this is a mistake or you have the requested information.)

@GimignanoF
Copy link

GimignanoF commented Aug 8, 2019

I don't know if you are still having issues, but i've managed to solve this by running this command
git config --global url."ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com".insteadOf "ssh://git-codecommit.${AWS_REGION}.amazonaws.com"
Wich added those lines in the global .gitconfig
[url "ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com"] insteadOf = ssh://git-codecommit.${AWS_REGION}.amazonaws.com

And so now i'm able to get Go packages from AWS CodeCommit using this command
go get git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/${REPOSITORY_NAME}.git

The ${AWS_REGION} placeholder has to be replaced with the actual region the repository is, for example in my case is EUWest1, so in the go get command i replaced it with "eu-west-1".
The ${REPOSITORY_NAME} placeholder is, well you guessed it, the name of the repository your GO package is in. For example, if your package is in a repository called "TestPackage" you must replace that placeholder with "TestPackage" as well (i don't know if it works having all lowercase too).

TL;DR

  • Run this command
    git config --global url."ssh://git@git-codecommit.${AWS_REGION}.amazonaws.com".insteadOf "ssh://git-codecommit.${AWS_REGION}.amazonaws.com"
    replacing ${AWS_REGION} with the Region the CodeCommit repository is in (for example eu-west-1)
  • Get the package via go get using this command
    go get git-codecommit.${AWS_REGION}.amazonaws.com/v1/repos/${REPOSITORY_NAME}.git
    where the ${AWS_REGION} placeholder is the same as before and the ${REPOSITORY_NAME} is the name of the repository (in your case should be myRepo). Don't forget to add the .git at the end, otherwise it won't work

@golang golang locked and limited conversation to collaborators Aug 7, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
FrozenDueToAge modules NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Development

No branches or pull requests

7 participants