crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

phacker · 2016-12-06T01:22:57Z

Please answer these questions before submitting your issue. Thanks!
Using MacBook Pro macOS 10.12.1

What version of Go are you using (`go version`)?

go1.7.4

What operating system and processor architecture are you using (`go env`)?

GOARCH="amd64"
GOBIN=""
GOEXE=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH=""
GORACE=""
GOROOT="/usr/local/go"
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
CC="clang"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/pb/vn9226r53sscpljzr0g3yfw40000gn/T/go-build062460274=/tmp/go-build -gno-record-gcc-switches -fno-common"
CXX="clang++"
CGO_ENABLED="1"

What did you do?

Compiling tip (80acfe9)
Using 1.7.4 standard install at /usr/local/go
$ printenv | grep ^GO
GOROOT_BOOTSTRAP=/usr/local/go
$ cd golang/go/src
$ ./all.bash
Pats-MBP:src path$ ./all.bash

Building Go bootstrap tool.

cmd/dist

Building Go toolchain using /usr/local/go.

bootstrap/cmd/internal/dwarf
bootstrap/cmd/internal/sys
bootstrap/cmd/asm/internal/flags
bootstrap/cmd/internal/bio
bootstrap/cmd/compile/internal/syntax
bootstrap/cmd/internal/gcprog
bootstrap/cmd/internal/obj
bootstrap/math/big
bootstrap/debug/pe
bootstrap/cmd/internal/obj/arm
bootstrap/cmd/internal/obj/arm64
bootstrap/cmd/internal/obj/mips
bootstrap/cmd/internal/obj/ppc64
bootstrap/cmd/internal/obj/s390x
bootstrap/cmd/internal/obj/x86
bootstrap/cmd/asm/internal/lex
bootstrap/cmd/link/internal/ld
bootstrap/cmd/asm/internal/arch
bootstrap/cmd/compile/internal/ssa
bootstrap/cmd/asm/internal/asm
bootstrap/cmd/asm
bootstrap/cmd/link/internal/arm
bootstrap/cmd/link/internal/amd64
bootstrap/cmd/link/internal/arm64
bootstrap/cmd/link/internal/mips
bootstrap/cmd/link/internal/mips64
bootstrap/cmd/link/internal/ppc64
bootstrap/cmd/link/internal/s390x
bootstrap/cmd/link/internal/x86
bootstrap/cmd/link
bootstrap/cmd/compile/internal/gc
bootstrap/cmd/compile/internal/amd64
bootstrap/cmd/compile/internal/arm
bootstrap/cmd/compile/internal/arm64
bootstrap/cmd/compile/internal/mips
bootstrap/cmd/compile/internal/mips64
bootstrap/cmd/compile/internal/ppc64
bootstrap/cmd/compile/internal/s390x
bootstrap/cmd/compile/internal/x86
bootstrap/cmd/compile

Building go_bootstrap for host, darwin/amd64.

runtime/internal/sys
runtime/internal/atomic
runtime
errors
sync/atomic
unicode/utf8
unicode/utf16
unicode
internal/syscall/windows/sysdll
math
internal/race
encoding
sync
syscall
io
internal/singleflight
hash
hash/adler32
bytes
strings
strconv
bufio
path
crypto
encoding/base64
reflect
crypto/sha1
internal/syscall/windows/registry
internal/syscall/windows
time
os
os/signal
encoding/binary
fmt
sort
path/filepath
container/heap
regexp/syntax
io/ioutil
context
debug/dwarf
encoding/json
log
go/token
net/url
compress/flate
text/template/parse
flag
os/exec
go/scanner
go/ast
regexp
compress/zlib
text/template
debug/macho
debug/elf
go/parser
go/doc
go/build
cmd/go

Building packages and commands for darwin/amd64.

runtime/internal/sys
runtime/internal/atomic
runtime
errors
unicode
internal/race
sync/atomic
unicode/utf8
math
sync
container/list
io
syscall
hash
hash/adler32
hash/crc32
strconv
bytes
strings
bufio
path
reflect
text/tabwriter
container/ring
crypto
crypto/subtle
crypto/internal/cipherhw
crypto/cipher
math/rand
crypto/sha512
crypto/aes
crypto/hmac
time
crypto/md5
crypto/rc4
crypto/sha1
crypto/sha256
encoding/base64
internal/nettrace
internal/singleflight
vendor/golang_org/x/crypto/poly1305
vendor/golang_org/x/crypto/curve25519
database/sql/internal
encoding
encoding/ascii85
encoding/base32
unicode/utf16
vendor/golang_org/x/text/transform
hash/crc64
hash/fnv
html
image/color
os
image
image/color/palette
internal/syscall/windows
vendor/golang_org/x/net/route
internal/syscall/windows/registry
internal/syscall/windows/sysdll
runtime/trace
image/internal/imageutil
math/cmplx
image/draw
image/jpeg
os/signal
runtime/race
cmd/compile/internal/test
cmd/vet/internal/whitelist
fmt
encoding/binary
sort
path/filepath
regexp/syntax
cmd/internal/sys
compress/bzip2
io/ioutil
container/heap
crypto/des
encoding/pem
vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20
runtime/debug
vendor/golang_org/x/crypto/chacha20poly1305
cmd/internal/dwarf
flag
log
debug/dwarf
compress/flate
debug/gosym
cmd/internal/obj
debug/plan9obj
cmd/vendor/golang.org/x/arch/arm/armasm
debug/macho
debug/pe
cmd/vendor/golang.org/x/arch/ppc64/ppc64asm
cmd/vendor/golang.org/x/arch/x86/x86asm
cmd/internal/goobj
compress/zlib
debug/elf
regexp
archive/tar
archive/zip
compress/gzip
compress/lzw
context
math/big
cmd/internal/objfile
encoding/hex
go/token
os/exec
go/scanner
go/ast
cmd/addr2line
database/sql/driver
database/sql
go/parser
go/printer
encoding/csv
encoding/gob
crypto/dsa
crypto/elliptic
encoding/asn1
crypto/rand
cmd/cgo
crypto/rsa
crypto/ecdsa
crypto/x509/pkix
encoding/json
encoding/xml
vendor/golang_org/x/net/http2/hpack
vendor/golang_org/x/net/idna
vendor/golang_org/x/text/unicode/norm
vendor/golang_org/x/text/width
mime
mime/quotedprintable
net/http/internal
net/url
text/template/parse
go/constant
go/format
text/scanner
image/gif
go/types
image/png
text/template
index/suffixarray
runtime/cgo
internal/pprof/profile
go/doc
html/template
go/build
net
testing
internal/trace
internal/testenv
runtime/pprof/internal/protopprof
net/internal/socktest
runtime/pprof
os/user
plugin
go/internal/gccgoimporter
go/internal/gcimporter
testing/internal/testdeps
testing/iotest
testing/quick
cmd/api
go/importer
cmd/internal/obj/arm
cmd/internal/obj/arm64
cmd/internal/obj/mips
cmd/internal/obj/ppc64
cmd/internal/obj/s390x
cmd/internal/obj/x86
cmd/asm/internal/flags
cmd/asm/internal/lex
cmd/internal/bio
cmd/compile/internal/syntax
cmd/internal/gcprog
crypto/x509
vendor/golang_org/x/net/lex/httplex
net/textproto
mime/multipart
log/syslog
net/mail
cmd/asm/internal/arch
cmd/compile/internal/ssa
cmd/asm/internal/asm
cmd/internal/browser
cmd/cover
cmd/asm
cmd/dist
cmd/doc
crypto/tls
cmd/fix
cmd/gofmt
net/http/httptrace
net/http
net/smtp
cmd/link/internal/ld
cmd/nm
cmd/objdump
cmd/pack
cmd/pprof/internal/plugin
cmd/pprof/internal/report
cmd/pprof/internal/svg
cmd/pprof/internal/tempfile
cmd/pprof/internal/commands
cmd/pprof/internal/driver
cmd/pprof/internal/symbolizer
cmd/pprof/internal/symbolz
cmd/vet/internal/cfg
cmd/vet
cmd/link/internal/amd64
expvar
net/http/cgi
net/http/cookiejar
net/http/httptest
net/http/fcgi
net/http/httputil
net/http/pprof
net/rpc
cmd/go
net/rpc/jsonrpc
cmd/link/internal/arm
cmd/link/internal/arm64
cmd/link/internal/mips
cmd/link/internal/mips64
cmd/link/internal/ppc64
cmd/link/internal/s390x
cmd/link/internal/x86
cmd/pprof/internal/fetch
cmd/link
cmd/pprof
cmd/trace
cmd/compile/internal/gc
cmd/compile/internal/amd64
cmd/compile/internal/arm
cmd/compile/internal/arm64
cmd/compile/internal/mips
cmd/compile/internal/mips64
cmd/compile/internal/ppc64
cmd/compile/internal/s390x
cmd/compile/internal/x86
cmd/compile

Testing packages.

ok archive/tar 0.027s
ok archive/zip 1.271s
ok bufio 0.060s
ok bytes 0.081s
ok compress/bzip2 0.058s
ok compress/flate 0.681s
ok compress/gzip 0.017s
ok compress/lzw 0.011s
ok compress/zlib 0.020s
ok container/heap 0.011s
ok container/list 0.010s
ok container/ring 0.022s
ok context 1.040s
ok crypto/aes 0.033s
ok crypto/cipher 0.028s
ok crypto/des 0.015s
ok crypto/dsa 0.011s
ok crypto/ecdsa 0.066s
ok crypto/elliptic 0.032s
ok crypto/hmac 0.006s
ok crypto/md5 0.014s
ok crypto/rand 0.028s
ok crypto/rc4 0.124s
ok crypto/rsa 0.086s
ok crypto/sha1 0.028s
ok crypto/sha256 0.009s
ok crypto/sha512 0.012s
ok crypto/subtle 0.010s
ok crypto/tls 1.229s
--- FAIL: TestSystemRoots (1.46s)
root_darwin_test.go:32: got 172 roots
root_darwin_test.go:32: got 132 roots
root_darwin_test.go:34: want at least 150 system roots, have 132
FAIL
FAIL crypto/x509 2.215s
ok database/sql 0.087s
ok database/sql/driver 0.008s
ok debug/dwarf 0.014s
ok debug/elf 0.029s
ok debug/gosym 0.243s
ok debug/macho 0.012s
ok debug/pe 0.012s
ok debug/plan9obj 0.010s
ok encoding/ascii85 0.010s
ok encoding/asn1 0.011s
ok encoding/base32 0.010s
ok encoding/base64 0.009s
ok encoding/binary 0.010s
ok encoding/csv 0.011s
ok encoding/gob 0.043s
ok encoding/hex 0.011s
ok encoding/json 0.385s
ok encoding/pem 0.019s
ok encoding/xml 0.016s
ok errors 0.009s
ok expvar 0.012s
ok flag 0.010s
ok fmt 0.103s
ok go/ast 0.013s
ok go/build 0.091s
ok go/constant 0.026s
ok go/doc 0.045s
ok go/format 0.013s
ok go/internal/gccgoimporter 0.010s
ok go/internal/gcimporter 0.517s
ok go/parser 0.035s
ok go/printer 0.336s
ok go/scanner 0.010s
ok go/token 0.026s
ok go/types 0.673s
ok hash/adler32 0.012s
ok hash/crc32 0.012s
ok hash/crc64 0.009s
ok hash/fnv 0.011s
ok html 0.008s
ok html/template 0.069s
ok image 0.111s
ok image/color 0.058s
ok image/draw 0.062s
ok image/gif 0.074s
ok image/jpeg 0.125s
ok image/png 0.035s
ok index/suffixarray 0.015s
ok internal/pprof/profile 0.009s
ok internal/singleflight 0.019s
ok internal/trace 0.394s
ok io 0.031s
ok io/ioutil 0.011s
ok log 0.012s
ok log/syslog 1.253s
ok math 0.011s
ok math/big 1.534s
ok math/cmplx 0.008s
ok math/rand 0.054s
ok mime 0.012s
ok mime/multipart 0.240s
ok mime/quotedprintable 0.088s
ok net 4.106s
ok net/http 5.470s
ok net/http/cgi 0.508s
ok net/http/cookiejar 0.015s
ok net/http/fcgi 0.013s
ok net/http/httptest 0.015s
ok net/http/httptrace 0.012s
ok net/http/httputil 0.033s
ok net/http/internal 0.008s
ok net/internal/socktest 0.010s
ok net/mail 0.011s
ok net/rpc 0.023s
ok net/rpc/jsonrpc 0.015s
ok net/smtp 0.020s
ok net/textproto 0.014s
ok net/url 0.016s
ok os 0.524s
ok os/exec 0.434s
ok os/signal 4.307s
ok os/user 0.009s
ok path 0.008s
ok path/filepath 0.041s
ok reflect 0.049s
ok regexp 0.098s
ok regexp/syntax 0.226s
ok runtime 16.145s
ok runtime/debug 0.010s
ok runtime/internal/atomic 0.103s
ok runtime/internal/sys 0.008s
ok runtime/pprof 1.594s
ok runtime/pprof/internal/protopprof 0.011s
ok runtime/trace 1.615s
ok sort 0.063s
ok strconv 0.500s
ok strings 0.105s
ok sync 0.214s
ok sync/atomic 0.029s
ok syscall 0.079s
ok testing 1.469s
ok testing/quick 0.091s
ok text/scanner 0.012s
ok text/tabwriter 0.211s
ok text/template 0.408s
ok text/template/parse 0.013s
ok time 2.434s
ok unicode 0.011s
ok unicode/utf16 0.009s
ok unicode/utf8 0.010s
ok vendor/golang_org/x/crypto/chacha20poly1305 0.266s
ok vendor/golang_org/x/crypto/chacha20poly1305/internal/chacha20 0.007s
ok vendor/golang_org/x/crypto/curve25519 0.023s
ok vendor/golang_org/x/crypto/poly1305 0.009s
ok vendor/golang_org/x/net/http2/hpack 0.010s
ok vendor/golang_org/x/net/idna 0.011s
ok vendor/golang_org/x/net/lex/httplex 0.009s
ok vendor/golang_org/x/net/route 0.022s
ok cmd/addr2line 0.531s
ok cmd/api 0.013s
ok cmd/asm/internal/asm 0.165s
ok cmd/asm/internal/lex 0.012s
ok cmd/compile 4.633s
ok cmd/compile/internal/gc 8.041s
ok cmd/compile/internal/ssa 0.120s
ok cmd/compile/internal/syntax 0.013s
ok cmd/compile/internal/test 0.009s [no tests to run]
ok cmd/cover 0.875s
ok cmd/doc 0.026s
ok cmd/fix 0.015s
ok cmd/go 36.364s
ok cmd/gofmt 0.034s
ok cmd/internal/goobj 0.008s
ok cmd/internal/obj 0.011s
ok cmd/internal/obj/arm64 0.008s
ok cmd/internal/obj/x86 0.010s
ok cmd/link 0.010s
ok cmd/nm 0.791s
ok cmd/objdump 1.189s
ok cmd/pack 1.486s
ok cmd/trace 0.014s
ok cmd/vendor/golang.org/x/arch/arm/armasm 0.011s
ok cmd/vendor/golang.org/x/arch/ppc64/ppc64asm 0.010s
ok cmd/vendor/golang.org/x/arch/x86/x86asm 0.209s
ok cmd/vet 1.750s
ok cmd/vet/internal/cfg 0.009s
2016/12/05 17:19:25 Failed: exit status 1

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.
A link on play.golang.org is best.

What did you expect to see?

Success

What did you see instead?

ok crypto/subtle 0.012s
ok crypto/tls 1.389s
--- FAIL: TestSystemRoots (1.28s)
root_darwin_test.go:32: got 172 roots
root_darwin_test.go:32: got 149 roots
root_darwin_test.go:34: want at least 150 system roots, have 149
FAIL
FAIL crypto/x509 2.081s

The text was updated successfully, but these errors were encountered:

quentinmit · 2016-12-06T01:37:45Z

Do you have a number of CAs disabled in Keychain Access? I guess we can slightly lower the threshold.

pathacker · 2016-12-06T03:43:56Z

Brand new mac. Nothing disabled in keychain that I can see.

…

On Mon, Dec 5, 2016 at 5:37 PM Quentin Smith ***@***.***> wrote: Do you have a number of CAs disabled in Keychain Access? I guess we can slightly lower the threshold. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#18203 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAGk3CwwmFbxZvoeU3IlYOeklygZSmnIks5rFLxrgaJpZM4LE6mP> .

bradfitz · 2016-12-06T04:22:20Z

@quentinmit, leaving for you to lower the threshold.

cookieo9 · 2016-12-06T08:53:16Z

I have a similar issue, my case:

root_darwin_test.go:34: want at least 150 system roots, have 145

In my case I've added certificates (to support delve, a debugger), although not necessarily root certificates. I've certainly not removed them. My machine was upgraded to Sierra (10.12) from Mavericks (10.10) so that might explain the difference there.

Has Apple invalidated some certificates recently? It might have something to do with: https://support.apple.com/en-us/HT202858

gopherbot · 2016-12-07T16:25:18Z

CL https://golang.org/cl/34019 mentions this issue.

quentinmit · 2016-12-07T19:24:00Z

@bradfitz @rsc
I'm worried that this is not as simple as lowering the threshold, which is why I didn't just send a CL to do that. My Sierra machine reports 169 roots and my El Capitan machine reports 175 roots.

The error report is about the second number printed by TestSystemRoots, which is the number of root CAs reported by the nocgo codepath. That codepath only looks at certificates provided by Apple, so the only way that number should differ should be if the user marks a cert as untrusted. The users here have reported that they didn't disable any certs, so that leaves me scratching my head.

phacker · 2016-12-07T19:33:10Z

How did you get your Sierra machine to report the number of roots? Is it the same as my Sierra laptop?

…

On Wed, Dec 7, 2016 at 11:24 AM Quentin Smith ***@***.***> wrote: Reopened #18203 <#18203>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18203 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH3GITm7aGyMxsiaEou4rjGuJ5PBuc-ks5rFwf1gaJpZM4LE6mP> .

phacker · 2016-12-07T19:34:04Z

Sorry. too fast on the trigger. I'm happy to report the number of roots listed on my laptop using the same mechanism you used.

…

On Wed, Dec 7, 2016 at 11:32 AM Pat Hacker ***@***.***> wrote: How did you get your Sierra machine to report the number of roots? Is it the same as my Sierra laptop? On Wed, Dec 7, 2016 at 11:24 AM Quentin Smith ***@***.***> wrote: Reopened #18203 <#18203>. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#18203 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH3GITm7aGyMxsiaEou4rjGuJ5PBuc-ks5rFwf1gaJpZM4LE6mP> .

quentinmit · 2016-12-07T19:48:41Z

@phacker
I'm just referring to the number of roots that Go logs, when you do

go test -v -run TestSystemRoots crypto/x509

Your log above says it found 132 roots. Actually, that's very weird - you also pasted a log that says it found 149 roots in the "What did you see instead?" section.

To eliminate any problems with Go, let's go back to the underlying data. It's stored in /System/Library/Keychains/SystemRootCertificates.keychain, so you should be able to count Apple's roots with:

security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | grep BEGIN | wc -l

This does not take into account disabled certificates. On my El Capitan machine, I get 189 certificates in the keychain, but when Go checks if they're trusted it ends up seeing a consistent 175 roots. If you run the "go test" command above, do you get a different number of roots each time?

phacker · 2016-12-07T19:55:11Z

$ go test -v -run TestSystemRoots crypto/x509 === RUN TestSystemRoots --- PASS: TestSystemRoots (0.95s) root_darwin_test.go:32: got 172 roots root_darwin_test.go:32: got 135 roots PASS ok crypto/x509 0.961s $ security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | grep BEGIN | wc -l 173 $ go test -v -run TestSystemRoots crypto/x509 === RUN TestSystemRoots --- PASS: TestSystemRoots (0.91s) root_darwin_test.go:32: got 172 roots root_darwin_test.go:32: got 148 roots PASS ok crypto/x509 0.925s $

…

On Wed, Dec 7, 2016 at 11:49 AM Quentin Smith ***@***.***> wrote: @phacker <https://github.com/phacker> I'm just referring to the number of roots that Go logs, when you do go test -v -run TestSystemRoots crypto/x509 Your log above says it found 132 roots. Actually, that's very weird - you also pasted a log that says it found 149 roots in the "What did you see instead?" section. To eliminate any problems with Go, let's go back to the underlying data. It's stored in /System/Library/Keychains/SystemRootCertificates.keychain, so you should be able to count Apple's roots with: security find-certificate -a -p /System/Library/Keychains/SystemRootCertificates.keychain | grep BEGIN | wc -l This does not take into account disabled certificates. On my El Capitan machine, I get 189 certificates in the keychain, but when Go checks if they're trusted it ends up seeing a consistent 175 roots. If you run the "go test" command above, do you get a different number of roots each time? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18203 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH3GGaZce52PEzaAdI1AVjLnXL9N6gxks5rFw24gaJpZM4LE6mP> .

quentinmit · 2016-12-07T20:04:17Z

So your machine has 173 roots of unknown trust, and you're seeing a random number of trusted roots less than that every time you run the test.

Well, that's fun. :)

Can you apply this patch and rerun the test to see if we get any useful debugging output?

diff --git a/src/crypto/x509/root_darwin.go b/src/crypto/x509/root_darwin.go
index 59b303d..dff1593 100644
--- a/src/crypto/x509/root_darwin.go
+++ b/src/crypto/x509/root_darwin.go
@@ -93,7 +93,8 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) {
                cmd = exec.Command("/usr/bin/security", "verify-cert", "-c", "/dev/stdin", "-l")
                cmd.Stdin = bytes.NewReader(data)
        }
-       if cmd.Run() == nil {
+       output, err := cmd.CombinedOutput()
+       if err == nil {
                // Non-zero exit means untrusted
                cert, err := ParseCertificate(block.Bytes)
                if err != nil {
@@ -101,7 +102,9 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) {
                }
 
                add(cert)
+               return
        }
+       fmt.Fprintf(os.Stderr, "verify-cert failed: %v\n%s\n", err, output)
 }

phacker · 2016-12-07T21:17:20Z

$ go test -v -run TestSystemRoots crypto/x509 === RUN TestSystemRoots verify-cert failed: exit status 1 Cert Verify Result: CSSMERR_TP_CERT_EXPIRED verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 Cert Verify Result: CSSMERR_TP_CERT_EXPIRED verify-cert failed: exit status 1 Cert Verify Result: CSSMERR_TP_CERT_EXPIRED verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid. verify-cert failed: exit status 1 SecCertificateCreateFromData: One or more parameters passed to a function were not valid.

…

--- PASS: TestSystemRoots (1.02s) root_darwin_test.go:32: got 172 roots root_darwin_test.go:32: got 149 roots PASS ok crypto/x509 1.035s $

On Wed, Dec 7, 2016 at 12:05 PM Quentin Smith ***@***.***> wrote: So your machine has 173 roots of unknown trust, and you're seeing a random number of trusted roots less than that every time you run the test. Well, that's fun. :) Can you apply this patch and rerun the test to see if we get any useful debugging output? diff --git a/src/crypto/x509/root_darwin.go b/src/crypto/x509/root_darwin.go index 59b303d..dff1593 100644 --- a/src/crypto/x509/root_darwin.go +++ b/src/crypto/x509/root_darwin.go @@ -93,7 +93,8 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) { cmd = exec.Command("/usr/bin/security", "verify-cert", "-c", "/dev/stdin", "-l") cmd.Stdin = bytes.NewReader(data) } - if cmd.Run() == nil { + output, err := cmd.CombinedOutput() + if err == nil { // Non-zero exit means untrusted cert, err := ParseCertificate(block.Bytes) if err != nil { @@ -101,7 +102,9 @@ func verifyCertWithSystem(block *pem.Block, add func(*Certificate)) { } add(cert) + return } + fmt.Fprintf(os.Stderr, "verify-cert failed: %v\n%s\n", err, output) } — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#18203 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAH3GH4WU1a-rnGbyNSY3zOItJbsFvxgks5rFxFpgaJpZM4LE6mP> .

bradfitz · 2016-12-08T22:12:30Z

@quentinmit, you planning on doing something here?

quentinmit · 2016-12-08T22:13:31Z

@bradfitz Good question. I don't know what the next step is on this bug; Apple's error message isn't exactly helpful. We can move to 1.8Maybe though.

quentinmit · 2016-12-12T20:07:56Z

@phacker We have another thing we'd like you to try, since you're able to reproduce this. Can you change needsTmpFile() in src/crypto/x509/root_darwin.go to unconditionally "return true" and see if you can reproduce the bug?

I'm afraid I've been completely unable to reproduce this on every Mac I have access to, both Sierra and El Capitan.

pathacker · 2016-12-12T21:46:21Z

Tip currently passes on my mac Sierra. Tip has changed x509/root_darwin_test.go TestSystemRoots from wanting 150 roots to wanting 100 roots. Changing this back to 150 causes my system to fail when doing ./all.bash Changing x509/root_darwin.go needsTmpFiles by adding the line return true as the first line of the function causes the build to pass even when wanting 150 roots. Pat

bradfitz · 2016-12-12T22:16:53Z

@quentinmit, yeah, sounds like the piping to stdin from a bytes.Reader is flaky. Is Apple's verify-cert code open source? I don't know which parts are & aren't. I'm curious if there's a bug in how they read from stdin.

quentinmit · 2016-12-12T22:32:22Z

@bradfitz Yeah, it's ostensibly open source. The entry point is at https://opensource.apple.com/source/Security/Security-57740.20.22/SecurityTool/verify_cert.c.auto.html

gopherbot · 2016-12-15T05:55:16Z

CL https://golang.org/cl/34389 mentions this issue.

bradfitz · 2016-12-16T23:43:36Z

See https://golang.org/cl/34389 for a new strategy that doesn't take 3.5 seconds to load certs.

/cc @mitchellh (could you test?)

cblecker · 2017-01-17T02:16:54Z

Hi @bradfitz --
Sorry for commenting on a closed issue, but I'm not sure where it would be more appropriate to (I'm not really involved in the golang community). I've seen this issue pop up another place. On Jan 11, the Google Cloud SDK team included a new version of kubectl with the google-cloud-sdk. This new version is compiled with go1.7.4, rather than the previous release that was compiled with go1.7.1.

This combination of kubectl compiled under go1.7.4 + macOS Sierra hits this bug, causing timeouts. I've seen mentions of it helm/helm#1749 and https://kubernetes.slack.com/archives/kubernetes-users/p1484587693022364, as well as encountered it myself. These issues seem to go away when using the version of kubectl compiled with Homebrew that is currently patching in the fix (Homebrew/homebrew-core#8628).

I'm not sure if there is a plan to release another 1.7.x go version before 1.8 drops next month, but I wanted to let you know that this bug is impacting users out in the wild.

If there is a better way or place to report this, then please let me know (and again my apologies).

minux · 2017-01-17T02:20:06Z

A better fix for software distributors is to enable cgo to avoid hitting this problem.

bradfitz · 2017-01-17T17:30:28Z

@cblecker, ugh. Please file a new bug and reference this one. (Comments on closed issues are where action items go to die.) No promises that we'll do a 1.7.5, but if we do, we'll consider it.

gopherbot · 2017-01-24T21:15:13Z

CL https://golang.org/cl/35634 mentions this issue.

…in root cert discovery Backporting Go 1.8's fix to #18203 Fixes #18688 --- Piping into security verify-cert only worked on macOS Sierra, and was flaky for unknown reasons. Users reported that the number of trusted root certs stopped randomly jumping around once they switched to using verify-cert against files on disk instead of /dev/stdin. But even using "security verify-cert" on 150-200 certs took too long. It took 3.5 seconds on my machine. More than 4 goroutines hitting verify-cert didn't help much, and soon started to hurt instead. New strategy, from comments in the code: // 1. Run "security trust-settings-export" and "security // trust-settings-export -d" to discover the set of certs with some // user-tweaked trusy policy. We're too lazy to parse the XML (at // least at this stage of Go 1.8) to understand what the trust // policy actually is. We just learn that there is _some_ policy. // // 2. Run "security find-certificate" to dump the list of system root // CAs in PEM format. // // 3. For each dumped cert, conditionally verify it with "security // verify-cert" if that cert was in the set discovered in Step 1. // Without the Step 1 optimization, running "security verify-cert" // 150-200 times takes 3.5 seconds. With the optimization, the // whole process takes about 180 milliseconds with 1 untrusted root // CA. (Compared to 110ms in the cgo path) Change-Id: I79737d9f2cb9b020ba297a326d4d31d68c7e9fee Reviewed-on: https://go-review.googlesource.com/35634 Reviewed-by: Joe Tsai <thebrokentoaster@gmail.com>

bradfitz changed the title ~~crypto/x509 TestSystemRoots~~ crypto/x509: TestSystemRoots Dec 6, 2016

bradfitz assigned quentinmit Dec 6, 2016

bradfitz added this to the Go1.8 milestone Dec 6, 2016

bradfitz added the Testing label Dec 6, 2016

quentinmit added the NeedsFix label Dec 6, 2016

quentinmit changed the title ~~crypto/x509: TestSystemRoots~~ crypto/x509: TestSystemRoots finds 0<n<150 certificates Dec 6, 2016

odeke-em mentioned this issue Dec 7, 2016

crypto/x509: root_darwin_test.go fails #18224

Closed

bradfitz assigned bradfitz and unassigned quentinmit Dec 7, 2016

gopherbot closed this as completed in be8a6fd Dec 7, 2016

quentinmit reopened this Dec 7, 2016

quentinmit removed the Testing label Dec 7, 2016

bradfitz assigned quentinmit and unassigned bradfitz Dec 8, 2016

quentinmit modified the milestones: Go1.8Maybe, Go1.8 Dec 8, 2016

bradfitz modified the milestones: Go1.8, Go1.8Maybe Dec 16, 2016

bradfitz assigned bradfitz and unassigned quentinmit Dec 16, 2016

bradfitz changed the title ~~crypto/x509: TestSystemRoots finds 0<n<150 certificates~~ crypto/x509: TestSystemRoots flaky & slow in non-cgo case Dec 16, 2016

bradfitz mentioned this issue Dec 16, 2016

crypto/x509: honor OS X certificate trust settings #18141

Closed

gopherbot closed this as completed in 3357daa Dec 20, 2016

mwarkentin mentioned this issue Jan 8, 2017

Terraform TLS Handshake Timeout errors Homebrew/homebrew-core#8576

Closed

3 tasks

cblecker mentioned this issue Jan 14, 2017

Intermittent x509: certificate signed by unknown authority error helm/helm#1749

Closed

cblecker mentioned this issue Jan 17, 2017

crypto/x509: certificate validation errors with go1.7.4 binaries + macOS Sierra #18688

Closed

fabianofranz mentioned this issue Apr 18, 2017

Upgrade golang-1.7 to 1.7.5 openshift/origin#13803

Merged

jhunt mentioned this issue Nov 17, 2017

Pulling system root CAs to validate doesn't work against the released copy of safe egen/safe#113

Closed

golang locked and limited conversation to collaborators Jan 24, 2018

gopherbot added the FrozenDueToAge label Jan 24, 2018

FiloSottile added the OS-Darwin label Apr 13, 2020

rsc unassigned bradfitz Jun 23, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

phacker commented Dec 6, 2016

quentinmit commented Dec 6, 2016

pathacker commented Dec 6, 2016 via email

bradfitz commented Dec 6, 2016

cookieo9 commented Dec 6, 2016

gopherbot commented Dec 7, 2016

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

phacker commented Dec 7, 2016 via email

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

bradfitz commented Dec 8, 2016

quentinmit commented Dec 8, 2016

quentinmit commented Dec 12, 2016

pathacker commented Dec 12, 2016 via email

bradfitz commented Dec 12, 2016

quentinmit commented Dec 12, 2016

gopherbot commented Dec 15, 2016

bradfitz commented Dec 16, 2016

cblecker commented Jan 17, 2017

minux commented Jan 17, 2017 via email

bradfitz commented Jan 17, 2017

gopherbot commented Jan 24, 2017

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

crypto/x509: TestSystemRoots flaky & slow in non-cgo case #18203

Comments

phacker commented Dec 6, 2016

What version of Go are you using (go version)?

What operating system and processor architecture are you using (go env)?

What did you do?

Building Go bootstrap tool.

Building Go toolchain using /usr/local/go.

Building go_bootstrap for host, darwin/amd64.

Building packages and commands for darwin/amd64.

Testing packages.

What did you expect to see?

What did you see instead?

quentinmit commented Dec 6, 2016

pathacker commented Dec 6, 2016 via email

bradfitz commented Dec 6, 2016

cookieo9 commented Dec 6, 2016

gopherbot commented Dec 7, 2016

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

phacker commented Dec 7, 2016 via email

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

quentinmit commented Dec 7, 2016

phacker commented Dec 7, 2016 via email

bradfitz commented Dec 8, 2016

quentinmit commented Dec 8, 2016

quentinmit commented Dec 12, 2016

pathacker commented Dec 12, 2016 via email

bradfitz commented Dec 12, 2016

quentinmit commented Dec 12, 2016

gopherbot commented Dec 15, 2016

bradfitz commented Dec 16, 2016

cblecker commented Jan 17, 2017

minux commented Jan 17, 2017 via email

bradfitz commented Jan 17, 2017

gopherbot commented Jan 24, 2017

What version of Go are you using (`go version`)?

What operating system and processor architecture are you using (`go env`)?