Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

x/vgo: vgo removed -insecure flag #25434

Closed
saibing opened this issue May 17, 2018 · 14 comments
Closed

x/vgo: vgo removed -insecure flag #25434

saibing opened this issue May 17, 2018 · 14 comments
Milestone

Comments

@saibing
Copy link

saibing commented May 17, 2018

What version of Go are you using (go version)?

go version go1.10.2 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/baas/.cache/go-build"
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/baas/git/go"
GORACE=""
GOROOT="/home/baas/my/go"
GOTMPDIR=""
GOTOOLDIR="/home/baas/my/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build608113082=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ curl -I http://rnd-github.huawei.com/db/parser/sql?go-get=1
HTTP/1.1 404 Not Found
Via: 1.1 172.18.32.117 (McAfee Web Gateway 7.6.2.14.0.23766)
Date: Thu, 17 May 2018 03:47:45 GMT
Server: GitHub.com
Status: 404 Not Found
X-Runtime: 0.006361
Content-Type: text/html; charset=utf-8
X-Request-Id: 715f56762358f8dbcf8968484163ceb3
Content-Length: 129318
X-Frame-Options: deny
Proxy-Connection: Keep-Alive
X-XSS-Protection: 1; mode=block
X-GitHub-Request-Id: 0f75ee93-d875-48d1-8dc7-8bcfa529656f
X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'; base-uri 'self'; block-all-mixed-content; child-src 'self'; connect-src 'self' http://rnd-github.huawei.com https://status.github.com rnd-github.huawei.com; font-src 'self'; form-action 'self' rnd-github.huawei.com; frame-ancestors 'none'; frame-src 'self'; img-src * data:; media-src 'none'; script-src 'self'; style-src 'unsafe-inline' 'self'
Connection: close

$ vgo get -insecure rnd-github.huawei.com/db/parser/sql

What did you expect to see?

get the rnd-github.huawei.com/db/parser/sql from github enterprise

What did you see instead?

flag provided but not defined: -insecure
usage: get [build flags] [packages]
Run 'go help get' for details.
@gopherbot gopherbot added this to the vgo milestone May 17, 2018
@kardianos kardianos changed the title x/vgo: not support http schema with github enterprise x/vgo: vgo removed -insecure flag May 20, 2018
@kardianos
Copy link
Contributor

@rsc Was the removal of the -insecure flag intentional? Should it stay removed?

@kardianos
Copy link
Contributor

Insecure was probably removed because many commands can now hit remote servers. Adding insecure flags to all such commands is probably not great idea.

Perhaps if this remains a requirement we could introduce a new environment variable that takes a list of host prefixes to whitelist.

GO_WHITELIST_ALLOW_INSECURE=internal.mycorp.local/vcs:vcs.test.local

@as
Copy link
Contributor

as commented May 27, 2018

I would prefer if this was a flag. I would like to see it with the ps command as that is easier to audit. Environment variables are significantly harder to trace in this manner.

@rsc
Copy link
Contributor

rsc commented Jun 6, 2018

This is the first report I've seen of GitHub Enterprise running over HTTP instead of HTTPS. Is it really not available under HTTPS in your environment?

One workaround is to put something in your ~/.gitconfig to tell git to rewrite https://your.server to http://your.server.

@saibing
Copy link
Author

saibing commented Jun 7, 2018

@rsc

Thank you for your answer very much.

However, this method does not solve the problem, because vgo does not read .gitconfig file, it still access https://rnd-github.huawei.com/db/parser?go-get=1.

baas@SZX1000428520:~/git/go/src/sql$ git config --get url.http://rnd-github.com/.insteadof
https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo get rnd-github.huawei.com/db/parser
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo get rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: finding rnd-github.huawei.com/go/otto v1.0.0
FindRepo: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/go/otto: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
vgo get: Get https://rnd-github.huawei.com/go/otto?go-get=1: cannotconnect
baas@SZX1000428520:~/git/go/src/sql$

@myitcv
Copy link
Member

myitcv commented Jun 7, 2018

It looks like your git config isn't quite right (missing a colon). Here's an example:

git config url.http://localhost:8080/:.insteadof https://github.com/

Notice the : before the .insteadof.

Also, you'll need to ensure that you don't have old configs that are potentially clashing:

git config -l | grep -i insteadof

But like @rsc asked, do you really not have https available?

@saibing
Copy link
Author

saibing commented Jun 7, 2018

@myitcv

Thank you very much.

I tested it, this does not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep -i insteadof
url.http://rnd-github.com/:.insteadof=https://rnd-github.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect
vgo: rnd-github.huawei.com/db/parser: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

For the follow problem, I have used nginx agent to solve this problem:)

But like @rsc asked, do you really not have https available?

@saibing
Copy link
Author

saibing commented Jun 7, 2018

@rsc @myitcv

Thank you for your support, I have use a nginx proxy to support https

@saibing saibing closed this as completed Jun 7, 2018
@myitcv
Copy link
Member

myitcv commented Jun 7, 2018

Just for the record, it looks like the insteadof config didn't work because you were using the incorrect domain. vgo was trying to resolve rnd-github.huawei.com yet your config was for rnd-github.com.

So you would need to have the following:

git config url.http://rnd-github.huawei.com/:.insteadof https://rnd-github.huawei.com/

I have use a nginx proxy to support https

Glad you have a solution.

@saibing
Copy link
Author

saibing commented Jun 7, 2018

@myitcv

I feel sorry for my mistake

I tested it again, but it still can not solve the problem

baas@SZX1000428520:~/git/go/src/sql$ git config -l | grep .insteadof
url.http://rnd-github.huawei.com/:.insteadof=https://rnd-github.huawei.com/
baas@SZX1000428520:~/git/go/src/sql$ vgo build
vgo: finding rnd-github.huawei.com/db/parser v0.0.0-20180125184044-316ea1d6f45f
FindRepo: Get https://rnd-github.huawei.com/db/parser?go-get=1: cannotconnect

I found that vgo does not use .gitconfig's url map:

func lookupCustomDomain(path string) (Repo, error) {
	var body io.ReadCloser
	err := web.Get(
		"https://"+path+"?go-get=1",
		web.Non200OK(),
		web.Body(&body),
	)
	if body != nil {
		defer body.Close()
	}
        //...
}

@myitcv
Copy link
Member

myitcv commented Jun 7, 2018

@saibing are you using the latest commit of vgo?

Looks like you are using an old version, pre #24915

Can you check:

go get -u golang.org/x/vgo
(command cd $(go list -f "{{.Dir}}" golang.org/x/vgo); git rev-parse HEAD)

outputs:

6a94eb3b5ccc04453d2fb45c23641e5993118068

for me.

@saibing
Copy link
Author

saibing commented Jun 7, 2018

@myitcv

me too:)

baas@SZX1000428520:~/git/go/src$ (command cd $(go list -f "{{.Dir}}" github.com/golang/vgo); git rev-parse HEAD)
6a94eb3b5ccc04453d2fb45c23641e5993118068

@myitcv
Copy link
Member

myitcv commented Jun 7, 2018

Sorry, I've also missed something obvious here. rnd-github.huawei.com is a custom domain. Hence the issue we're seeing here happens before git is called; because it's the resolution of https://rnd-github.huawei.com/db/parser?go-get=1 that is failing here. This step is required so that vgo can understand where the underlying VCS repo lives, and which tool to use (git, svn etc).

I'll defer to @rsc on whether we want to align this with the go tool or not.

@AlekSi
Copy link
Contributor

AlekSi commented Jun 7, 2018

Do you really not have https available?

If a company runs GitHub Enterprise or something similar behind VPN, they typically do not use HTTPS because setting it up is a hassle. For example, if they use their own custom domain like .internal it is not possible to get a "real" (trusted by major CAs) certificate, and creating their own CA and make it trusted on all hosts is painful.

So -insecure flag should ignore certificate check errors (if HTTPS is used with a self-signed certificate or certificate is issued by a custom CA), and should try HTTP is HTTPS is not available.

@golang golang locked and limited conversation to collaborators Jun 7, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

7 participants