Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

crypto/x509: certificate validation errors with go1.7.4 binaries + macOS Sierra #18688

Closed
cblecker opened this issue Jan 17, 2017 · 10 comments
Closed

Comments

@cblecker
Copy link

Please answer these questions before submitting your issue. Thanks!

What version of Go are you using (go version)?

go version go1.7.4 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/opt/gopath"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/T/go-build124753553=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

On Jan 11, the Google Cloud SDK team included a new version of kubectl with the google-cloud-sdk. This new version is compiled with go1.7.4, rather than the previous release that was compiled with go1.7.1.

This combination of kubectl compiled under go1.7.4 + macOS Sierra seems to hit #18203, causing the certificates to report as invalid. I've seen mentions of it helm/helm#1749 and https://kubernetes.slack.com/archives/kubernetes-users/p1484587693022364, as well as encountered it myself. These issues seem to go away when using the version of kubectl compiled with Homebrew that is currently patching in the fix (Homebrew/homebrew-core#8628).

I'm not sure if there is a plan to release another 1.7.x go version before 1.8 drops next month, but I wanted to let you know that this bug is impacting users out in the wild.

What did you expect to see?

Darwin binaries compiled with go1.7.4 and run on macOS Sierra seem to hit the bug described in #18203. This causes problems when validating TLS certificates.

What did you see instead?

kubectl (go binary to control Kubernetes clusters from the command line) specificly reports Unable to connect to the server: Post https://accounts.google.com/o/oauth2/token: x509: certificate signed by unknown authority. This error is eliminated when using a version of kubectl that was compiled with 3357daa patched in.

References

#18141
#18203
helm/helm#1749
Homebrew/homebrew-core#8628

@bradfitz bradfitz added this to the Go1.7.5 milestone Jan 17, 2017
@bradfitz
Copy link
Contributor

I think your bug report is misleading. You say:

CGO_ENABLED="1"

But from what I understand, this bug is specifically about CGO_ENABLED=0 binaries.

Can you confirm that the kubectl binary in question was built without cgo support?

@cblecker
Copy link
Author

My local go environment is CGO_ENABLED, but the binary in question is pre-compiled and distributed by the Google Cloud SDK team. I'm not sure if there is a way to tell from the binary itself if CGO was used in it's compilation.

@bradfitz
Copy link
Contributor

Use go tool nm on the kubectl binary. If it was built without cgo, you'll see the execSecurityRoots symbol in the binary:

$ go tool nm hello.cgo  | grep execSecurityRoots
$ go tool nm hello.cgo0  | grep execSecurityRoots
   56330 T crypto/x509.execSecurityRoots

@cblecker
Copy link
Author

That was easy!

$ google-cloud-sdk/bin/kubectl version --client
Client Version: version.Info{Major:"1", Minor:"5", GitVersion:"v1.5.1", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"clean", BuildDate:"2016-12-14T00:57:05Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
$ go tool nm google-cloud-sdk/bin/kubectl | grep execSecurityRoots
  dde4a0 T crypto/x509.execSecurityRoots
  dea6e0 T crypto/x509.execSecurityRoots.func1
  dea790 T crypto/x509.execSecurityRoots.func2
 1473ed0 R crypto/x509.execSecurityRoots.func2.f

@cblecker
Copy link
Author

cblecker commented Jan 17, 2017

This is odd.. with my local go environment, I recompiled kubectl from source, and I'm still getting execSecurityRoots in go tool nm:

$ make generated_files 
+++ [0117 11:18:00] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:01] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:02] Building go targets for darwin/amd64:
    cmd/libs/go2idl/deepcopy-gen
+++ [0117 11:18:08] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:08] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:08] Building go targets for darwin/amd64:
    cmd/libs/go2idl/defaulter-gen
+++ [0117 11:18:13] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:13] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:14] Building go targets for darwin/amd64:
    cmd/libs/go2idl/conversion-gen
+++ [0117 11:18:20] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:18:20] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:18:21] Building go targets for darwin/amd64:
    cmd/libs/go2idl/openapi-gen
$ make kubectl
+++ [0117 11:19:01] Building the toolchain targets:
    k8s.io/kubernetes/hack/cmd/teststale
    k8s.io/kubernetes/vendor/github.com/jteeuwen/go-bindata/go-bindata
+++ [0117 11:19:01] Generating bindata:
    test/e2e/framework/gobindata_util.go
+++ [0117 11:19:01] Building go targets for darwin/amd64:
    cmd/kubectl
$ _output/bin/kubectl version --client
Client Version: version.Info{Major:"1", Minor:"5+", GitVersion:"v1.5.1-dirty", GitCommit:"82450d03cb057bab0950214ef122b67c83fb11df", GitTreeState:"dirty", BuildDate:"2017-01-17T19:19:01Z", GoVersion:"go1.7.4", Compiler:"gc", Platform:"darwin/amd64"}
$ go tool nm  _output/bin/kubectl | grep execSecurityRoots
  dde4a0 T crypto/x509.execSecurityRoots
  deafc0 T crypto/x509.execSecurityRoots.func1
 14753a8 R crypto/x509.execSecurityRoots.func1.f
$ go env
GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/opt/gopath"
GORACE=""
GOROOT="/usr/local/opt/go/libexec"
GOTOOLDIR="/usr/local/opt/go/libexec/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/lw/b5k2lvx53j98z_c0z8cyk_bcgjcp8_/T/go-build362149565=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"
$ 

@bradfitz
Copy link
Contributor

@cblecker, but what is the Makefile doing? It's probably disabling cgo.

@cblecker
Copy link
Author

Yeah, had to dig in a bunch of layers, but kubectl is handled as a static binary in the kubernetes project, and is compiled with CGO_ENABLED=0 here:
https://github.com/kubernetes/kubernetes/blob/master/hack/lib/golang.sh#L534

@titanous
Copy link
Member

I have also run into the same problem with CGO_ENABLED=0 and my own (non-Kubernetes) binaries with the same Go/macOS versions.

@gopherbot
Copy link

CL https://golang.org/cl/35634 mentions this issue.

gopherbot pushed a commit that referenced this issue Jan 25, 2017
…in root cert discovery

Backporting Go 1.8's fix to #18203
Fixes #18688

---

Piping into security verify-cert only worked on macOS Sierra, and was
flaky for unknown reasons. Users reported that the number of trusted
root certs stopped randomly jumping around once they switched to using
verify-cert against files on disk instead of /dev/stdin.

But even using "security verify-cert" on 150-200 certs took too
long. It took 3.5 seconds on my machine. More than 4 goroutines
hitting verify-cert didn't help much, and soon started to hurt
instead.

New strategy, from comments in the code:

// 1. Run "security trust-settings-export" and "security
//    trust-settings-export -d" to discover the set of certs with some
//    user-tweaked trusy policy. We're too lazy to parse the XML (at
//    least at this stage of Go 1.8) to understand what the trust
//    policy actually is. We just learn that there is _some_ policy.
//
// 2. Run "security find-certificate" to dump the list of system root
//    CAs in PEM format.
//
// 3. For each dumped cert, conditionally verify it with "security
//    verify-cert" if that cert was in the set discovered in Step 1.
//    Without the Step 1 optimization, running "security verify-cert"
//    150-200 times takes 3.5 seconds. With the optimization, the
//    whole process takes about 180 milliseconds with 1 untrusted root
//    CA. (Compared to 110ms in the cgo path)

Change-Id: I79737d9f2cb9b020ba297a326d4d31d68c7e9fee
Reviewed-on: https://go-review.googlesource.com/35634
Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>
@bradfitz
Copy link
Contributor

Submitted to 1.7 branch.

k8s-github-robot pushed a commit to kubernetes/kubernetes that referenced this issue Feb 23, 2017
Automatic merge from submit-queue (batch tested with PRs 41812, 41665, 40007, 41281, 41771)

Bump golang versions to 1.7.5

**What this PR does / why we need it**: While #41636 might not make it in until 1.7, this would bump current golang versions from 1.7.4 to 1.7.5 to integrate the fixes from that patch version. This would include, among other things, a fix to ensure cross-built binaries for darwin don't have certificate validation errors (golang/go#18688)

**Which issue this PR fixes** *(optional, in `fixes #<issue number>(, fixes #<issue_number>, ...)` format, will close that issue when PR gets merged)*: none

**Special notes for your reviewer**:

**Release note**:

```release-note
Upgrade golang versions to 1.7.5
```
@golang golang locked and limited conversation to collaborators Jan 26, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants