Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

cmd/go: go get -insecure fetches over http instead of https #34427

Closed
kyroy opened this issue Sep 20, 2019 · 15 comments
Closed

cmd/go: go get -insecure fetches over http instead of https #34427

kyroy opened this issue Sep 20, 2019 · 15 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Milestone
Go1.14

Comments

@kyroy
Copy link
Contributor

kyroy commented Sep 20, 2019

What version of Go are you using (go version)?

$ go version
go version go1.13 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output
$ go env
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/root/.cache/go-build"
GOENV="/root/.config/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build880420340=/tmp/go-build -gno-record-gcc-switches"

What did you do?

$ docker run  -ti --rm golang:1.13 bash
$ mkdir test && cd test && go mod init test

$ go get -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)

$ go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (https fetch: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: x509: certificate signed by unknown authority)

What did you expect to see?

Fetch the information via https but ignore the invalid certificate

What did you see instead?

Go get used http.
@mvdan
Copy link
Member

mvdan commented Sep 20, 2019 •
edited

This is clearly documented:

$ go help get
[...]
The -insecure flag permits fetching from repositories and resolving
custom domains using insecure schemes such as HTTP. Use with caution.
[...]

Are you proposing that we change what the flag does? If so, wouldn't that break existing users? Imagine if a repository or custom domain doesn't support HTTPS, only HTTP.

@kyroy
Copy link
Contributor Author

kyroy commented Sep 20, 2019

I see, sorry.

How can I then ignore the certificate? Is there any way?

@mvdan
Copy link
Member

mvdan commented Sep 20, 2019

Duplicate of #13197? If https with disabled TLS certificate checks isn't being tried first, this is probably a regression or bug.

/cc @bcmills @jayconrod

@mvdan
Copy link
Member

mvdan commented Sep 20, 2019

Also, please provide the output of one of the go get lines with -v, to see what http/https urls are being tried.

@kyroy
Copy link
Contributor Author

kyroy commented Sep 20, 2019

-v provides exactly the same output

$ go get -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)

@tmthrgd
Copy link
Contributor

tmthrgd commented Sep 20, 2019

@kyroy I believe @mvdan meant the -x flag.

@mvdan
Copy link
Member

mvdan commented Sep 20, 2019

Ah, it looks like -x is the right flag nowadays. -v used to give the URLs being queried, but now it's only -x.

@kyroy
Copy link
Contributor Author

kyroy commented Sep 20, 2019

Without GOPRIVATE

$ go get -x -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: finding github.xxxx.xxxxx.corp/myorg v0.0.0-20190903123812-3090d622918c
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/go/@v/v0.0.0-20190903123812-3090d622918c.info
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/@v/v0.0.0-20190903123812-3090d622918c.info
go: finding github.xxxx.xxxxx.corp v0.0.0-20190903123812-3090d622918c
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/@v/v0.0.0-20190903123812-3090d622918c.info
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/go/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (7.790s)
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (8.441s)
# get https://proxy.golang.org/github.xxxx.xxxxx.corp/myorg/@v/v0.0.0-20190903123812-3090d622918c.info: 410 Gone (8.443s)
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)

With

$ export GOPRIVATE=github.xxxx.xxxxx.corp
$ go get -x -v -insecure github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get https://github.xxxx.xxxxx.corp/myorg?go-get=1
# get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get https://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg?go-get=1
# get //github.xxxx.xxxxx.corp/?go-get=1: Get https://github.xxxx.xxxxx.corp/?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/?go-get=1
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get https://github.xxxx.xxxxx.corp/myorg/go?go-get=1: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
# get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1
# get //github.xxxx.xxxxx.corp/myorg?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/?go-get=1: Get http://github.xxxx.xxxxx.corp/?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
# get //github.xxxx.xxxxx.corp/myorg/go?go-get=1: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout
go get github.xxxx.xxxxx.corp/myorg/go@v0.0.0-20190903123812-3090d622918c: unrecognized import path "github.xxxx.xxxxx.corp/myorg/go" (http/https fetch: Get http://github.xxxx.xxxxx.corp/myorg/go?go-get=1: dial tcp 10.XX.XX.XX:80: i/o timeout)
$ curl -k https://github.xxxx.xxxxx.corp/?go-get=1
<html><body>You are being <a href="https://github.xxxx.xxxxx.corp/repositories">redirected</a>.</body></html>

@toothrot toothrot added the NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. label Sep 20, 2019
@toothrot toothrot added this to the Go1.14 milestone Sep 20, 2019
@bcmills
Copy link
Contributor

bcmills commented Sep 20, 2019

@kyroy, note that the go command follows redirects (curl -kL not just curl -k).

From the trace you've provided, it appears that the go command with -insecure is correctly trying https URLs before falling back to http. You just happen to get the http URL in the error message because that's the last one it tried.

@bcmills
Copy link
Contributor

bcmills commented Sep 20, 2019

Please try curl -kL https://github.xxxx.xxxxx.corp/myorg/go?go-get=1 and see where you end up.

@bcmills bcmills added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 20, 2019
@kyroy
Copy link
Contributor Author

kyroy commented Sep 23, 2019

Hi, sorry for the delay and thanks for the investigation help :)

$ curl -kL https://github.xxxx.xxxxx.corp/myorg/go?go-get=1

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">



  <title>Go remote import path metadata</title>
  <meta name="go-import" content="github.xxxx.xxxxx.corp/myorg/go git https://github.xxxx.xxxxx.corp/myorg/go.git">



  <link rel="mask-icon" href="https://github.xxxx.xxxxx.corp/pinned-octocat.svg" color="#000000">
  <link rel="icon" type="image/x-icon" class="js-site-favicon" href="https://github.xxxx.xxxxx.corp/favicon-ent.ico">

<meta name="theme-color" content="#1e2327">




  <link rel="manifest" href="/manifest.json" crossOrigin="use-credentials">

  </head>

  <body>

  <!-- Metadata for Go remote import path -->

  </body>
</html>

@FiloSottile FiloSottile removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Sep 23, 2019
@bcmills
Copy link
Contributor

bcmills commented Sep 24, 2019

@kyroy, that doesn't really explain why the connection would be timing out. Is it possible that the server was overloaded or otherwise had very high latency?

@kyroy
Copy link
Contributor Author

kyroy commented Sep 25, 2019

@bcmills Indeed I found out that there has been a bug on the infrastructure in our availability zone that the DNS server was not configured properly. Hence, there was a >5s latency.

Thanks for helping. I am not sure if the error message can be improved. Otherwise, this can be closed :)

(out of scope of this issue)
Another thing that I have experienced a long time ago was that -insecure is not fully enough. You also need to set GIT_SSL_NO_VERIFY. Is there a reason for this / cant this be included?

$ go get -insecure github.xxxx.xxxxx.corp/myorg/go
go get github.xxxx.xxxxx.corp/myorg/go: git ls-remote -q https://github.xxxx.xxxxx.corp/myorg/go.git in /go/pkg/mod/cache/vcs/1e0d9b889f3416a56ea37502ad1137f6723e61f8260c10aaf3fb8c45d44204fe: exit status 128:
	fatal: unable to access 'https://github.xxxx.xxxxx.corp/myorg/go.git/': server certificate verification failed. CAfile: none CRLfile: none

$ GIT_SSL_NO_VERIFY=1 go get -insecure github.xxxx.xxxxx.corp/myorg/go
go: finding github.xxxx.xxxxx.corp/myorg/go latest
go: downloading github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c
go: extracting github.xxxx.xxxxx.corp/myorg/go v0.0.0-20190903123812-3090d622918c

@bcmills
Copy link
Contributor

bcmills commented Sep 26, 2019

(out of scope of this issue)

Please file that as a separate issue so that we don't lose track of it. Thanks.

@bcmills bcmills closed this as completed Sep 26, 2019
@kyroy
Copy link
Contributor Author

kyroy commented Sep 27, 2019

Done. Opened #34568

@golang golang locked and limited conversation to collaborators Sep 26, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one.
Projects
None yet
Milestone
Go1.14
Development

No branches or pull requests
7 participants
@toothrot @tmthrgd @FiloSottile @mvdan @kyroy @bcmills @gopherbot