What version of Go are you using (go version)?

The issue reproduces (with variants of symptoms) on go 1.18.2, 1.19.6 and 1.20.5

What operating system and processor architecture are you using (go env)?

Apple M1.
(The issue does not reproduce on linux or apple x86).

$ go env
GO111MODULE=""
GOARCH="arm64"
GOBIN=""
GOCACHE="/Users/kena/Library/Caches/go-build"
GOENV="/Users/kena/Library/Application Support/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="arm64"
GOHOSTOS="darwin"
GOINSECURE=""
GOMODCACHE="/Users/kena/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/kena/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/opt/homebrew/opt/go/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/opt/homebrew/opt/go/libexec/pkg/tool/darwin_arm64"
GOVCS=""
GOVERSION="go1.20.5"
GCCGO="gccgo"
AR="ar"
CC="cc"
CXX="c++"
CGO_ENABLED="1"
GOMOD="/Users/kena/cockroach/go.mod"
GOWORK=""
CGO_CFLAGS="-O2 -g"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-O2 -g"
CGO_FFLAGS="-O2 -g"
CGO_LDFLAGS="-O2 -g"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -arch arm64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/zj/9kw0pgws50v2m_jj3qshxyt00000gq/T/go-build4040752470=/tmp/go-build -gno-record-gcc-switches -fno-common"

What did you do?

• Using CockroachDB (v22.2 or v23.1) built with go 1.18.2 or 1.19.6 on Apple M1, starting up the binary crashes in crypto/x509 with the stack trace below

  Stack trace

  
  

   
  SIGILL: illegal instruction
  PC=0x7ff80e31d52f m=0 sigcode=1
  instruction bytes: 0xf 0xb 0x48 0x8d 0x5 0xe1 0xe6 0x18 0x0 0x48 0x89 0x5 0x19 0xb7 0x74 0x41
  goroutine 0 [idle]:
  
  crypto/x509/internal/macos.syscall(0xc003276750?, 0x0?, 0x10037f820?, 0xc004e17260?, 0x10000d98b?, 0x10492f8c0?, 0xc003276750?)
  
  GOROOT/src/runtime/sys_darwin.go:100 +0x95 fp=0xc004e17210 sp=0xc004e17170 pc=0x10006d935
  
  crypto/x509/internal/macos.CFRelease(0xc004e172b8?)
  
  GOROOT/src/crypto/x509/internal/macos/corefoundation.go:156 +0x34 fp=0xc004e17258 sp=0xc004e17210 pc=0x10037ea14
  
  crypto/x509/internal/macos.ReleaseCFArray(0xc004e172d0?)
  
  GOROOT/src/crypto/x509/internal/macos/corefoundation.go:218 +0x2c fp=0xc004e17288 sp=0xc004e17258 pc=0x10037ecec
  
  crypto/x509.(*Certificate).systemVerify.func2()
  
  GOROOT/src/crypto/x509/root_darwin.go:36 +0x26 fp=0xc004e172a0 sp=0xc004e17288 pc=0x10038f706
  
  runtime.deferreturn()
  
  GOROOT/src/runtime/panic.go:476 +0x33 fp=0xc004e172e0 sp=0xc004e172a0 pc=0x10003a813
  
  crypto/x509.(*Certificate).systemVerify(0xc0007ae000, 0xc004e176c0)
  
  GOROOT/src/crypto/x509/root_darwin.go:42 +0x7b8 fp=0xc004e17580 sp=0xc004e172e0 pc=0x10038eff8
  
  crypto/x509.(*Certificate).Verify(0xc0007ae000, {{0xc002c64180, 0x18}, 0xc000e883c0, 0x0, {0xc11cf1e0dfaf0db0, 0x7cdd0503, 0x10944b400}, {0x0, 0x0, ...}, ...})
  
  GOROOT/src/crypto/x509/verify.go:768 +0x4e7 fp=0xc004e176c0 sp=0xc004e17580 pc=0x100393e47
  
  crypto/tls.(*Conn).verifyServerCertificate(0xc002dd1880, {0xc0006b1c80, 0x3, 0x4})
  
  GOROOT/src/crypto/tls/handshake_client.go:885 +0x679 fp=0xc004e17948 sp=0xc004e176c0 pc=0x1003c5f39
  
  

• To reproduce the issue the binary must be called via .//../cockroach .... If the binary is called via ./cockroach, ./../cockroach or other paths that do not include //.., the issue does not reproduce.

• The issue can be further reproduced using a small repro program in go 1.20.5 (also included below).

  Test program `test.go` to reproduce

  
  

  package main
  import (
  
  "bytes"
  
  "context"
  
  "fmt"
  
  "io"
  
  "net/http"
  
  "os"
  
  )
  func post(
  
  ctx context.Context, url, contentType string, body io.Reader,
  
  ) (resp *http.Response, err error) {
  
  req, err := http.NewRequestWithContext(ctx, "POST", url, body)
  
  if err != nil {
  
  return nil, err
  
  }
  
  req.Header.Set("Content-Type", contentType)
  
  var client http.Client
  
  return client.Do(req)
  
  }
  const url = https://register.cockroachdb.com/api/clusters/report
  func main() {
  
  ctx := context.Background()
  
  b := []byte("hello")
  
  res, err := post(ctx, url, "application/x-protobuf", bytes.NewReader(b))
  
  if err != nil {
  
  fmt.Println(err)
  
  os.Exit(1)
  
  }
  
  defer res.Body.Close()
  
  b, err = io.ReadAll(res.Body)
  
  if err != nil || res.StatusCode != http.StatusOK {
  
  fmt.Println(err)
  
  os.Exit(1)
  
  }
  
  fmt.Println(string(b))
  
  }
  
  

  Steps to reproduce:

  mkdir t && cd t   # this path is important, see last command below
  go mod init
  cp path-to-test-source/test.go .
  go build test.go
  .//../t/test
  Post "https://register.cockroachdb.com/api/clusters/report": tls: failed to verify certificate: SecPolicyCreateSSL error: 0
  

  If the program is ran via ./test it completes successfully.

  Also running via .//../t/test on linux does not repro the issue -- it's specific to macOS.

xref: cockroachdb/cockroach#105534

What did you expect to see?

The program initializes TLS successfully.

What did you see instead?

See above.