crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

joegrasse · 2017-01-27T16:12:23Z

What version of Go are you using (`go version`)?

go version go1.8rc3 linux/386

What operating system and processor architecture are you using (`go env`)?

GOARCH="386"
GOBIN=""
GOEXE=""
GOHOSTARCH="386"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/usr2/ps/isp/go"
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/linux_386"
GCCGO="gccgo"
GO386=""
CC="gcc"
GOGCCFLAGS="-fPIC -m32 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build005892966=/tmp/go-build"
CXX="g++"
CGO_ENABLED="1"
PKG_CONFIG="pkg-config"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"

What did you do?

go get -d -u <package location>

What did you expect to see?

Successful completion of go get

What did you see instead?

package <package location>: unrecognized import path "<package location>" (https fetch: Get https://<package_url>?go-get=1: x509: certificate signed by unknown authority)

Using go version go1.7.4 linux/386, this worked successfully. We are using our own CA, and the package_url is our own internal server.

The text was updated successfully, but these errors were encountered:

joegrasse · 2017-01-27T17:14:03Z

Doing more research looks like

go/src/crypto/x509/root_linux.go

Line 10 in ce64553

"/etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem", // CentOS/RHEL 7

was added in go1.8. On our server we have both /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem and /etc/pki/tls/certs/ca-bundle.crt. In go 1.7.4, it would find /etc/pki/tls/certs/ca-bundle.crt, which has the root certificate. In go 1.8rc3, it found /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem first, which doesn't have the root certificate.

Our server is CentOS 6.8.

bradfitz · 2017-01-27T17:22:03Z

If this were found during the Go 1.8 beta testing period, or even an earlier rc, it would've been easier to fix. It might be too late now. :-/

As a workaround, can you just add your certs to the first location?

Do other popular TLS clients read & merge all files?

/cc @agl

joegrasse · 2017-01-27T17:25:30Z

@bradfitz By default the openssl command was working correctly. It was able to verify the certificate.

bradfitz · 2017-01-27T17:26:48Z

But because it's looking at one location or merging multiple?

joegrasse · 2017-01-27T17:29:57Z

Here is a snippet from man verify.

"The process of ’looking up the issuers certificate’ itself involves a number of steps. In versions of OpenSSL before 0.9.5a the first certificate whose subject name matched the issuer of the current certificate was assumed to be the issuers certificate. In OpenSSL 0.9.6 and later all certificates whose subject name matches the issuer name of the current certificate are subject to further tests."

Which to me sounds like it keeps looking.

agl · 2017-01-27T17:48:20Z

I think that section of the man page is suggesting that it'll check multiple certificates from a file, as opposed to merging several files. Generally OpenSSL is configured with https://www.openssl.org/docs/man1.0.2/ssl/SSL_CTX_load_verify_locations.html which takes a single file and a single directory for hash lookups. Of course, OpenSSL is very complex and it's possible to make things happen differently and there's the possibility of distro-specific patches too.

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

joegrasse · 2017-01-27T18:05:01Z

Why do we think that the system OpenSSL isn't just looking at /etc/pki/tls/certs/ca-bundle.crt?

I not saying it doesn't, but with the change in go1.8, go isn't looking there by default.

I am still trying to research this, but it looks like in CentOS 6 (not sure what dot revision it was introduced) there is a legacy way and a new way to handle certs.

joegrasse · 2017-01-27T18:42:18Z

@bradfitz I am leaning toward this being a breaking change for CentOS 6, because I believe by default CentOS 6 uses the legacy method, which is /etc/pki/tls/certs/ca-bundle.crt. I am by no means an expert in this though.

agl · 2017-01-27T18:48:04Z

To clarify: does CentOS include both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem but expect that the former is the default?

bradfitz · 2017-01-27T18:50:28Z

If we got this bug report two months a fix would've been much easier.

Between go1.8beta1, go1.8beta2, go1.8rc1, go1.8rc2, and go1.8rc3 we make fewer and smaller and safer changes. We're basically not making changes at this point of the release cycle. We can't risk causing regressions for other crypto/x509 users. The fact that we only got one bug report about this (and so late) suggests this isn't an issue that will impact many people at least.

Unless there's a great diagnosis of what's happening and a super minimal and obviously-correct fix, you'll probably have to just modify both *.crt files, use a locally-patched Go, stay on Go 1.7, or wait for Go 1.9. Sorry.

joegrasse · 2017-01-27T18:52:28Z

@bradfitz I have already worked around the issue, I am more just worried about the CentOS 6 user that this might break. What population of them have tested go1.8?

joegrasse · 2017-01-27T18:54:29Z

To clarify: does CentOS include both /etc/pki/tls/certs/ca-bundle.crt and /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem but expect that the former is the default?

@agl I believe Yes and Yes. I don't have a "stock" CentOS 6 box though.

agl · 2017-01-27T18:58:03Z

It sounds like we could just flip the order of the two entries then, but I'd love to get some clarification from RedHat about their policy here. I'll see if I can reach someone who'll claim to know.

joegrasse · 2017-01-27T19:57:15Z

@bradfitz @agl I am wondering for easier discovery and because this doesn't just impact go get if the title should now be

crypto/x509: "certificate signed by unknown authority" regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change.

agl · 2017-01-27T23:14:43Z

(I've emailed @nmav to see if we can get word about what the "correct" answer is here.)

nmav · 2017-01-28T17:26:47Z

In RHEL7 the file to use for openssl apps is /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem [0]. ca-bundle.crt is a link to the first one.

In RHEL6 the primary file is ca-bundle.crt. (There, it is optional for an administrator to enable the rhel7 trust management, by running update-ca-trust enable. In that case tls-ca-bundle.pem and ca-bundle.crt should be the same file).

Based on the above I think using ca-bundle.crt seems like a safe choice.

[0]. http://www.unix.com/man-page/centos/8/update-ca-trust/

joegrasse · 2017-01-30T17:49:40Z

@agl As a side note, I have noticed the following.

CentOS 7

~$ ls -alh /etc/pki/tls/cert.pem
lrwxrwxrwx. 1 root root 49 Nov 29 02:53 /etc/pki/tls/cert.pem -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

CentOS 6

~$ ls -alh /etc/pki/tls/cert.pem
lrwxrwxrwx. 1 root root 19 Jun  8  2016 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt

Not sure if it would make sense to just point at /etc/pki/tls/cert.pem?

joegrasse · 2017-01-31T15:28:03Z

@bradfitz @agl It looks like ce64553 is the commit that caused this problem. It also looks like it was a fix for #15749. On CentOS 7 though, /etc/pki/tls/certs/ca-bundle.crt is linked to /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem. So, I think there was a different cause for the issue in #15749.

bradfitz · 2017-02-01T23:19:48Z

I defer to @agl to decide what to do here.

@agl, we ship Feb 14th. It'd be nice to have this done before then. There might be an -rc4.

gopherbot · 2017-02-02T01:25:33Z

CL https://golang.org/cl/36093 mentions this issue.

rsc · 2017-02-07T14:28:27Z

I agree that CL 36093 is too much for Go 1.8. Can we just move the new RHEL7 line down to the bottom of the preference list for Go 1.8?

ping @agl

rsc · 2017-02-07T17:03:59Z

Sent CL 36429 to reorder the list. I checked, and the line was added for #17549, which was "can't find any certs"; that is, on the system we were trying to fix, none of the listed files existed. On this CentOS 6 system, the existing files were better than the new file, but putting the new file earlier in the list broke things. Moving the new file to the end of the list should keep #17549 fixed and still fix this issue.

gopherbot · 2017-02-07T17:05:30Z

CL https://golang.org/cl/36429 mentions this issue.

agl · 2017-02-07T17:12:03Z

It's still unclear why /etc/pki/tls/certs/ca-bundle.crt didn't exist in #15749, but Russ's change should still take care of that so LGTM.

rsc · 2017-02-07T17:20:34Z

https://go-review.googlesource.com/36530 for cherry-pick to Go 1.8.

gopherbot · 2017-02-07T17:25:13Z

CL https://golang.org/cl/36530 mentions this issue.

We added CentOS 7's /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem to the list in response to #17549 - not being able to find any certs otherwise. Now we have #18813, where CentOS 6 apparently has both that file and /etc/pki/tls/certs/ca-bundle.crt, and the latter is complete while the former is not. Moving the new CentOS 7 file to the bottom of the list should fix both problems: the CentOS 7 system that didn't have any of the other files in the list will still find the new one, and existing systems will still keep using what they were using instead of preferring the new path that may or may not be complete on some systems. Fixes #18813. Change-Id: I5275ab67424b95e7210e14938d3e986c8caee0ba Reviewed-on: https://go-review.googlesource.com/36429 Run-TryBot: Russ Cox <rsc@golang.org> Reviewed-by: Adam Langley <agl@golang.org> Reviewed-on: https://go-review.googlesource.com/36530 TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org>

rsc · 2017-02-08T18:49:54Z

Closing. CL 36093 mentions this issue but is really about a broader thing. This specific issue should now be fixed entirely.

Add the ability to override the default file and directory from which certificates are loaded by setting the OpenSSL compatible environment variables: SSL_CERT_FILE, SSL_CERT_DIR. If the variables are set the default locations are not checked. Added new default file "/usr/local/etc/ssl/cert.pem" for FreeBSD. Certificates in the first valid location found for both file and directory are added, instead of only the first file location if a valid one was found, which is consistent with OpenSSL. Fixes #3905 Fixes #14022 Fixes #14311 Fixes #16920 Fixes #18813 - If user sets SSL_CERT_FILE. Change-Id: Ia24fb7c1c2ffff4338b4cf214bd040326ce27bb0 Reviewed-on: https://go-review.googlesource.com/36093 Reviewed-by: Brad Fitzpatrick <bradfitz@golang.org> Run-TryBot: Brad Fitzpatrick <bradfitz@golang.org> TryBot-Result: Gobot Gobot <gobot@golang.org>

joegrasse changed the title ~~cmd/go: Regression with go get of package~~ cmd/go: Regression with go get of package from go1.7.4 to go1.8rc3 Jan 27, 2017

bradfitz changed the title ~~cmd/go: Regression with go get of package from go1.7.4 to go1.8rc3~~ crypto/x509: go get regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change Jan 27, 2017

bradfitz added this to the Go1.8Maybe milestone Jan 27, 2017

bradfitz assigned agl Jan 27, 2017

bradfitz added the NeedsDecision Feedback is required from experts, contributors, and/or the community before a change can be made. label Jan 27, 2017

bradfitz changed the title ~~crypto/x509: go get regression from go1.7 to go1.8 on CentOS due to cert file lookup paths change~~ crypto/x509: path lookup order change affects CentOS from Go 1.7->8 Jan 27, 2017

rsc modified the milestones: Go1.8, Go1.8Maybe Feb 7, 2017

gopherbot closed this as completed in 1ead0bd Feb 7, 2017

rsc reopened this Feb 7, 2017

agl assigned rsc and unassigned agl Feb 8, 2017

rsc modified the milestones: Go1.9, Go1.8 Feb 8, 2017

rsc closed this as completed Feb 8, 2017

jefferai mentioned this issue Feb 28, 2017

Vault 0.6.5 "x509: certificate signed by unknown authority" hashicorp/vault#2415

Closed

golang locked and limited conversation to collaborators Feb 8, 2018

gopherbot added the FrozenDueToAge label Feb 8, 2018

rsc removed their assignment Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

joegrasse commented Jan 27, 2017

joegrasse commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

joegrasse commented Jan 27, 2017 •

edited

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017 •

edited

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

nmav commented Jan 28, 2017 •

edited

joegrasse commented Jan 30, 2017

joegrasse commented Jan 31, 2017

bradfitz commented Feb 1, 2017

gopherbot commented Feb 2, 2017

rsc commented Feb 7, 2017

rsc commented Feb 7, 2017

gopherbot commented Feb 7, 2017

agl commented Feb 7, 2017

rsc commented Feb 7, 2017

gopherbot commented Feb 7, 2017

rsc commented Feb 8, 2017

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

crypto/x509: path lookup order change affects CentOS from Go 1.7->8 #18813

Comments

joegrasse commented Jan 27, 2017

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

What did you expect to see?

What did you see instead?

joegrasse commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

joegrasse commented Jan 27, 2017 • edited

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

bradfitz commented Jan 27, 2017

joegrasse commented Jan 27, 2017 • edited

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

joegrasse commented Jan 27, 2017

agl commented Jan 27, 2017

nmav commented Jan 28, 2017 • edited

joegrasse commented Jan 30, 2017

joegrasse commented Jan 31, 2017

bradfitz commented Feb 1, 2017

gopherbot commented Feb 2, 2017

rsc commented Feb 7, 2017

rsc commented Feb 7, 2017

gopherbot commented Feb 7, 2017

agl commented Feb 7, 2017

rsc commented Feb 7, 2017

gopherbot commented Feb 7, 2017

rsc commented Feb 8, 2017

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?

joegrasse commented Jan 27, 2017 •

edited

joegrasse commented Jan 27, 2017 •

edited

nmav commented Jan 28, 2017 •

edited