Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

crypto/x509: loadSystemRoots segfault when running under launchd #17972

Closed
nhooyr opened this issue Nov 18, 2016 · 13 comments
Closed

crypto/x509: loadSystemRoots segfault when running under launchd #17972

nhooyr opened this issue Nov 18, 2016 · 13 comments
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Milestone
Go1.9

Comments

@nhooyr
Copy link
Contributor

nhooyr commented Nov 18, 2016 •
edited

What version of Go are you using (go version)?

go version devel +b21743c Thu Nov 17 20:53:01 2016 +0000 darwin/amd64

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOOS="darwin"

With the following program https://play.golang.org/p/HzaMgW_1p4 and the following Launch Agent definition

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
	<dict>
		<key>Label</key>
		<string>playground</string>
		<key>ProgramArguments</key>
		<array>
			<string>/Users/nhooyr/code/gopath/bin/playground</string>
		</array>
		<key>RunAtLoad</key>
		<true/>
		<key>StandardErrorPath</key>
		<string>/tmp/playgroundLogs</string>
	</dict>
</plist>

I get

fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x127491f]

runtime stack:
runtime.throw(0x113f88e, 0x2a)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/panic.go:596 +0x95
runtime.sigpanic()
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/signal_unix.go:253 +0x2db

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x1001520, 0xc42009db18, 0xc42000e028)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/cgocall.go:130 +0xe2 fp=0xc42009dae8 sp=0xc42009daa8
crypto/x509._Cfunc_FetchPEMRoots(0xc42000e028, 0xc400000000)
	crypto/x509/_obj/_cgo_gotypes.go:100 +0x49 fp=0xc42009db18 sp=0xc42009dae8
crypto/x509.loadSystemRoots.func1(0xc42000e028, 0xc42000e028)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/crypto/x509/root_cgo_darwin.go:170 +0xbd fp=0xc42009db60 sp=0xc42009db18
crypto/x509.loadSystemRoots(0x0, 0x0, 0x0)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/crypto/x509/root_cgo_darwin.go:170 +0x15d fp=0xc42009dbd8 sp=0xc42009db60
crypto/x509.initSystemRoots()
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/crypto/x509/root.go:21 +0x26 fp=0xc42009dc10 sp=0xc42009dbd8
sync.(*Once).Do(0x11f37c0, 0x1141ee0)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/sync/once.go:44 +0xbe fp=0xc42009dc48 sp=0xc42009dc10
crypto/x509.systemRootsPool(0x9)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/crypto/x509/root.go:16 +0x39 fp=0xc42009dc68 sp=0xc42009dc48
crypto/x509.(*Certificate).Verify(0xc42009e000, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/crypto/x509/verify.go:287 +0x7e1 fp=0xc42009ded0 sp=0xc42009dc68
main.main()
	/Users/nhooyr/code/gopath/src/github.com/nhooyr/playground/main.go:41 +0x134 fp=0xc42009df88 sp=0xc42009ded0
runtime.main()
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/proc.go:185 +0x20a fp=0xc42009dfe0 sp=0xc42009df88
runtime.goexit()
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/asm_amd64.s:2184 +0x1 fp=0xc42009dfe8 sp=0xc42009dfe0

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
	/nix/store/c8dvzgg10k5svh5mrgkgdiikkzrbmf9g-go-1.8-dev/share/go/src/runtime/asm_amd64.s:2184 +0x1

in /tmp/playgroundLogs.

If I do not run the program as a Launch Agent under launchd, all works fine.
@nhooyr
Copy link
Contributor Author

nhooyr commented Nov 18, 2016

Seems related to #16508

@quentinmit
Copy link
Contributor

What version of OS X are you using?

@quentinmit quentinmit added NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. labels Nov 18, 2016
@quentinmit quentinmit added this to the Go1.8 milestone Nov 18, 2016
@nhooyr
Copy link
Contributor Author

nhooyr commented Nov 18, 2016

macOS Sierra 10.12.1

@quentinmit quentinmit removed the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Nov 18, 2016
@ianlancetaylor
Copy link
Contributor

The stack backtrace is showing that something in FetchPEMRoots, a C function defined in a cgo comment in crypto/x509/root_cgo_darwin.go, is dereferencing a NULL pointer. The NULL dereference could be in FetchPEMRoots or it could be in some function that it calls.

I assume this is easy to replicate. If so, could you run the program under gdb or lldb and try to get a backtrace of the program from the point of failure? That may help us see where the problem is in the C code. Thanks.

@nhooyr
Copy link
Contributor Author

nhooyr commented Nov 25, 2016 

(lldb) target create "/Users/nhooyr/code/gopath/bin/playground"
Current executable set to '/Users/nhooyr/code/gopath/bin/playground' (x86_64).
(lldb) run
Process 1400 stopped
* thread #1: tid = 0x5606, 0x000000000130a64f, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x000000000130a64f
->  0x130a64f: testb  $0x4, (%rax)
    0x130a652: je     0x130a662
    0x130a654: movq   0x48(%rax), %rax
    0x130a658: testq  %rax, %rax

Process 1400 launched: '/Users/nhooyr/code/gopath/bin/playground' (x86_64)
(lldb) bt
* thread #1: tid = 0x5606, 0x000000000130a64f, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x000000000130a64f
    frame #1: 0x000000000135f951
    frame #2: 0x000000000130a6b3
    frame #3: 0x000000000100144d playground`FetchPEMRoots + 461 at asm_amd64.s:122
    frame #4: 0x000000000100153d playground`_cgo_0e26f01c964f_Cfunc_FetchPEMRoots + 29 at asm_amd64.s:122
    frame #5: 0x000000000104ce50 playground`runtime.asmcgocall + 112 at asm_amd64.s:616
    frame #6: 0x00000000011d8b00 playground`runtime.work + 384
    frame #7: 0x00000000010f43c9 playground`crypto/x509._Cfunc_FetchPEMRoots(p0=0x000000c42000e028, r1=0) + 73 at _cgo_gotypes.go:100
    frame #8: 0x00000000010f48ed playground`crypto/x509.loadSystemRoots.func1(_cgo0=0x000000c42000e028, ~r1=536928296) + 189 at root_cgo_darwin.go:170
    frame #9: 0x00000000010f45dd playground`crypto/x509.loadSystemRoots(~r0=0x0000000000000000, ~r1=error @ 0x000000c42009fbe0) + 349 at root_cgo_darwin.go:170
    frame #10: 0x00000000010eafc6 playground`crypto/x509.initSystemRoots + 38 at root.go:21
    frame #11: 0x000000000105377e playground`sync.(o=0x00000000011f37c0, f=<no summary available>).Do + 190 at once.go:44
    frame #12: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #13: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #14: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #15: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #16: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #17: 0x00000000010f48ed playground`crypto/x509.loadSystemRoots.func1(_cgo0=0x000000c42000e028, ~r1=536928296) + 189 at root_cgo_darwin.go:170
    frame #18: 0x00000000010f45dd playground`crypto/x509.loadSystemRoots(~r0=0x0000000000000000, ~r1=error @ 0x000000c42009fbe0) + 349 at root_cgo_darwin.go:170
    frame #19: 0x00000000010eafc6 playground`crypto/x509.initSystemRoots + 38 at root.go:21
    frame #20: 0x000000000105377e playground`sync.(o=0x00000000011f37c0, f=<no summary available>).Do + 190 at once.go:44
    frame #21: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #22: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #23: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #24: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #25: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #26: 0x00000000010f45dd playground`crypto/x509.loadSystemRoots(~r0=0x0000000000000000, ~r1=error @ 0x000000c42009fbe0) + 349 at root_cgo_darwin.go:170
    frame #27: 0x00000000010eafc6 playground`crypto/x509.initSystemRoots + 38 at root.go:21
    frame #28: 0x000000000105377e playground`sync.(o=0x00000000011f37c0, f=<no summary available>).Do + 190 at once.go:44
    frame #29: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #30: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #31: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #32: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #33: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #34: 0x00000000010eafc6 playground`crypto/x509.initSystemRoots + 38 at root.go:21
    frame #35: 0x000000000105377e playground`sync.(o=0x00000000011f37c0, f=<no summary available>).Do + 190 at once.go:44
    frame #36: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #37: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #38: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #39: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #40: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #41: 0x000000000105377e playground`sync.(o=0x00000000011f37c0, f=<no summary available>).Do + 190 at once.go:44
    frame #42: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #43: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #44: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #45: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #46: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #47: 0x00000000010eaf79 playground`crypto/x509.systemRootsPool(~r0=0x0000000000000009) + 57 at root.go:16
    frame #48: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #49: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #50: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #51: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #52: 0x00000000010ec551 playground`crypto/x509.(c=0x000000c4200a0000, opts=crypto/x509.VerifyOptions @ 0x000000c42009fed8, chains=(len 842350780272, cap 17788283), err=error @ 0x000000c42009ff40).Verify + 2017 at verify.go:287
    frame #53: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #54: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #55: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #56: 0x00000000010f6f04 playground`main.main + 308 at main.go:41
    frame #57: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #58: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #59: 0x000000000102941a playground`runtime.main + 522 at proc.go:185
    frame #60: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184
    frame #61: 0x000000000104e0d1 playground`runtime.goexit + 1 at asm_amd64.s:2184

@ianlancetaylor
Copy link
Contributor

Thanks. From this we can see that it is not FetchPEMRoots itself that is doing the NULL dereference, but some C library function that it calls. Unfortunately that is as far as I can take this. I don't know specifically what is failing. Presumably this does not happen on most systems but I don't know what is different about your system.

@josharian
Copy link
Contributor

I can't reproduce on my macOS 10.12.1 system.

If we can figure out which library function is dying, that might provide some clues. Can you toss some printfs into FetchPEMRoots to find out? And then find out what the arguments are to the problematic call? Even if that doesn't point to the problem directly, it might provide enough clues to help lldb resolve the mystery pcs in the backtrace.

@bradfitz
Copy link
Contributor

bradfitz commented Dec 8, 2016

@nhooyr can you reproduce with the latest code at master?

Also, is this a regression from Go 1.7? (Does Go 1.7 work for you under launchd?)

@bradfitz bradfitz added the WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided. label Dec 8, 2016
@bradfitz bradfitz modified the milestones: Go1.8Maybe, Go1.8 Dec 8, 2016
@nhooyr
Copy link
Contributor Author

nhooyr commented Dec 9, 2016 •
edited

Running under Go 1.7.3 also segfaults.

(lldb) target create "/Users/nhooyr/code/gopath/bin/playground"
Current executable set to '/Users/nhooyr/code/gopath/bin/playground' (x86_64).
(lldb) run
Process 1475 stopped
* thread #1: tid = 0x2ef8d, 0x00000000002f791f, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
    frame #0: 0x00000000002f791f
->  0x2f791f: testb  $0x4, (%rax)
    0x2f7922: je     0x2f7932
    0x2f7924: movq   0x48(%rax), %rax
    0x2f7928: testq  %rax, %rax

Process 1475 launched: '/Users/nhooyr/code/gopath/bin/playground' (x86_64)
(lldb) bt
* thread #1: tid = 0x2ef8d, 0x00000000002f791f, stop reason = EXC_BAD_ACCESS (code=1, address=0x0)
  * frame #0: 0x00000000002f791f
    frame #1: 0x0000000000357941 Security`construction vtable for Security::CssmClient::ObjectImpl-in-Security::CssmClient::SSGroupImpl + 65
    frame #2: 0x00000000002f7983
    frame #3: 0x00000000000f2c0d libxml2.2.dylib`html40EntitiesTable + 605
    frame #4: 0x00000000000f2cfd libxml2.2.dylib`html40EntitiesTable + 845
    frame #5: 0x00000000000f2ce0 libxml2.2.dylib`html40EntitiesTable + 816
    frame #6: 0x00000000000523b0 libsystem_m.dylib
    frame #7: 0x000000000005da49 libsystem_network.dylib
    frame #8: 0x000000000005dd2b libsystem_network.dylib
    frame #9: 0x0000000000054ed6
    frame #10: 0x0000000000087e3b libcorecrypto.dylib`_ccdh_gp_rfc5114_MODP_2048_256 + 251
    frame #11: 0x0000000000000010 liblzma.5.dylib
    frame #12: 0x0000000000054e89
    frame #13: 0x0000000000056146 libdispatch.dylib
    frame #14: 0x000000000000212a libenergytrace.dylib
    frame #15: 0x000000000002a1e4 libxpc.dylib

I'll try @josharian's suggstion next.

@nhooyr
Copy link
Contributor Author

nhooyr commented Dec 9, 2016

Welp, seems to be working fine on master now. No idea what was wrong. Thanks!

@nhooyr nhooyr closed this as completed Dec 9, 2016
@nhooyr
Copy link
Contributor Author

nhooyr commented Dec 9, 2016

Actually, while it works under launchd, Go itself has stopped compiling under nix because of what I think is the same segfault. I'll update with details tomorrow.

@nhooyr nhooyr reopened this Dec 9, 2016
@bradfitz bradfitz modified the milestones: Go1.9, Go1.8Maybe Dec 9, 2016
@bradfitz
Copy link
Contributor

bradfitz commented Dec 9, 2016

I'm bumping this to Go 1.9 until we even know what we're talking about in this bug.

We have nothing reproducible and it's not clear whether x509 has a problem or the compiler has a problem or this is a nix issue or this is your machine only.

Most importantly, this isn't a regression from 1.7 (you said 1.7 also crashed), so this seems like 1.9 material.

@zimbatm zimbatm mentioned this issue Mar 20, 2017
Merged
7 tasks
@bradfitz
Copy link
Contributor

bradfitz commented Jul 6, 2017

Timed out.

@bradfitz bradfitz closed this as completed Jul 6, 2017
@golang golang locked and limited conversation to collaborators Jul 6, 2018
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
FrozenDueToAge NeedsInvestigation Someone must examine and confirm this is a valid issue and not a duplicate of an existing one. WaitingForInfo Issue is not actionable because of missing required information, which needs to be provided.
Projects
None yet
Milestone
Go1.9
Development

No branches or pull requests
6 participants
@bradfitz @josharian @quentinmit @ianlancetaylor @gopherbot @nhooyr